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This  volume  of  the  pamphlet  is  a  compendium  of  lessons  learned  reports 
on  significant  formal  source  selections  conducted  by  the  U.S.  Army 
Materiel  Command.  It  will  be  periodically  updated  by  adding  selected 
after  action  reports  from  future  Source  Selection  Evaluation  Boards. 
The  initial  printing  of  this  volume  includes  the  lessons  learned 
from  the  programs  listed  below.  Future  updates  of  this  volume  will 
include  a  comprehensive  list  of  after  action  reports  that  have  been 
selected  for  publication  in  this  pamphlet. 
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INTRODUCTION 


This  after-action  report  sununarizes  events  and  processes  involved  in 
performing  the  evaluation  of  proposals  for  the  T800-XX-800  Engine.  The 
acquisition  strategy  established  earlier  during  development  of  the  Request 
for  Proposal  (RFP  DAAJ09-84-R-A955)  was  implemented  as  described  in  the 
approved  evaluation  plan.  It  is  intended  that  this  document  provide 
highlights  of  the  evaluation  activity,  describe  the  impact  of  the  Army's 
new  way  of  doing  business,  and  recommend  improvements  that  can  be  applied 
to  future  procurements.  The  emphasis  is  on  the  source  selection  process, 
not  on  the  recommended  winners,  their  scores,  or  their  standings. 


Although  the  T800  is  considered  to  be  a  generic  power  plant,  its  first 
application  will  be  on  the  Army's  new  family  of  light  helicopters  (LHX) . 
Discussions  of  LHX  engine  requirements  are  provided  in  the  previous  RFP, 
its  after-action  report,  and  other  LHX  project  documents  and  will  not  be 
repeated  here.  However,  the  acquisition  strategy  and  overall  approach  are 
a  fallout  of  LHX  planning,  and  this  document  reflects  the  results  of 
early-on  decisions  coordinated  and  approved  at  the  Army's  highest  leader¬ 
ship  levels,  and  can  serve  as  the  last  step  in  the  audit  trail. 


The  evaluation  resulted  in  two  contract  awards  and  combined  many  new 
initiatives  (listed  below)  into  a  precedent-setting  procurement. 


o  Short,  concise  Request  for  Proposal. 


o  Requirements  defined  without  dictating  to  offerors  how  to  design, 
plan,  or  accomplish  tasks. 


o  Firm  fixed  price  contract  for  development. 


o  Contract  commitments  and  guaranteed  prices  that  carry  over  into 
production,  operation  and  support. 


o  Commitments  to  develop  alternate  sources  and  competition  throughout 
production  of  the  end  item  and  parts. 


o  Commitments  to  establish  strong  vendor  program  for  small 
bus i nesses . 


o  Definitions  of  contractor  liabilities  for  not  meeting  commitments 
to  guarantees  or  for  not  meeting  system  specification. 


o  Commitments  to  plans  for  Reliability  and  Maintainability  and 
Integrated  Logistics  Support  (RAM/ILS)  with  guarantees  to  redesign, 
retest,  requalify,  and  retrofit  without  increase  in  cost  until  successful 
completion  ot  FSD . 


o  Commitment  for  contractor  to  facilitize  at  no  direct  cost  to  the 
government.  It  must  be  noted  that  each  of  the  above  commitments,  initia¬ 
tives  and  changes  is  in  place  now  (up  front)  with  long-term  guarantees 


well  beyond  development  and  that  the  competitive  environment  has  been 
established  well  in  advance  of  qualification  and  production.  In  a  rever¬ 
sal  of  typical  past  practices  of  stating  goals  and  negotiating  prices 
later,  details  of  commitments  now  are  written  into  the  contracts. 

Figure  1  provides  an  overview  of  the  initiatives  to  serve  as  a  ready 
reference  in  reading  this  after-action  report.  Figure  2,  also  provided 
for  reference  purposes,  is  an  overview  of  the  strategy  applied  to  promote 
competition  throughout  the  life  cycle.  It  is  significant  to  note  that  tv^ 
engine  manufacturers  make  up  each  developer's  team.  One  team  consists  of 
AVCO-Lycoming  and  Pratt  and  Whitney,  and  the  other  consists  of  Garrett 
Turbine  Engine  Company  and  Allison  Gas  Turbine  Division  of  General  Motors. 
Each  team  will  compete  its  design  against  the  other  during  early  develop¬ 
ment  (over  about  a  3-year  period)  up  to  a  Preliminary  Flight  Rating  (PFR). 
At  that  time,  when  many  of  the  critical  requirements  and  strengths  will 
have  been  demonstrated,  the  government  will  down-select  to  one  team  based 
on  contract  performance,  other  notable  successes,  and  progress  toward 
commitments.  The  surviving  team  then  will  complete  the  developmeift  and 
early  production  of  the  same  engine  design.  These  remaining  two  manufac¬ 
turers  then  will  split  and  compete  with  each  other  for  full-rate  produc¬ 
tion  and  necessary  support,  using  guaranteed  prices  (already  negotiated) 
as  the  ceilings.  An  important  aspect  of  the  production  initiatives  is 
that  each  will  bid  its  price  to  the  government  independent  of  the  former 
team  relationship. 


SUMMARY 


During  1984,  the  Army  initiated  planning  and  developed  acquisition  strate¬ 
gy  for  procurement  of  the  T800-XX-800  Engine.  A  major  thrust  was  to 
establish  a  new  way  for  the  Army  to  conduct  business  with  the  industry. 

In  the  process,  many  firsts  were  defined,  along  with  many  changes  to 
previous  practices.  This  document  covers  the  entire  process  -  from 
development  of  the  Request  for  Proposal  (RFP)  to  the  completion  of  tasks 
performed  by  the  Source  Selection  Evaluation  Board  (SSEB) .  To  assist  the 
reader,  this  report  is  divided  into  sections  that  represent  steps  along 
the  audit  trail,  including  preparations  for  convening  and  operating  the 
SSEB,  the  evaluation  itself,  and  the  concluding  reports.  Throughout  the 
effort,  running  logs  were  kept  by  the  participants  so  that  information 
could  be  collected  and  consolidated  to  produce  this  after-action  report. 

The  innovations,  new  initiatives,  and  acquisition  strategy,  together, 
represent  a  considerable  departure  from  past  practices.  The  primary 
subjects  addressed  during  preparation  of  the  RFP  were  cost,  production 
competition,  and  product  performance.  In  turn,  the  focus  of  the  acquisi¬ 
tion  strategy  became  competitive  development  and  competitive  follow-on 
production.  With  these  overall  concepts  in  hand,  it  then  became  the  task 
of  this  SSEB  to  execute  plans  for  selecting  two  competing  teams  to  develop 
the  T800  to  a  point  where  the  design  that  offered  the  best  value  to  the 
government  could  be  selected  (about  3  years  from  the  date  of  this  report) 
for  continuation  into  qualification  and  production  phases. 

The  approach  was  implemented  via  the  RFP,  and  the  two  contractor  teams 
(AVCO/Pratt  &  Whitney  and  Garrett/Allison)  selected  after  the  evaluation 
will  compete  development  of  their  engine  designs  on  a  firm  fixed  price 
basis  with  up-front  commitments  to  guaranteed  production  prices  and 
operational  costs.  These  commitments  also  involve  demonstration  of 
performance  requirements  and  development  of  at  least  two  competitive 
sources  for  replacement  parts,  as  well  as  dual  sources  for  the  end  item  in 
production.  The  team  that  best  demonstrates  success  against  established 
requirements  will  be  chosen  to  complete  the  remainder  of  the  program. 

After  qualification  of  the  engine,  the  two  major  manufacturers  on  the 
winning  team  also  will  compete  with  each  other  during  production  to 
provide  the  same  design  at  the  best  value,  including  operating  and  support 
guarantees.  To  obtain  the  long-term  benefits,  these  departures  from  past 
practices  required  that  the  contractors  guarantee  their  cost  and  perfor¬ 
mance  now,  rather  than  cite  future  goals  and  negotiate  later.  A  summary 
of  the  new  initiatives  is  as  follows. 

COST 


o  Development  -  Firm  fixed  price  contract,  including  provisions  for 
necessary  corrective  redesign,  retest,  requalification,  and  retrofit,  at 
no  additional  cost. 

o  Production  -  Guaranteed  ceiling  prices  for  engines  with  government 
flexibility  to  vary  lot  quantities. 


o  Operational  Support  -  Guaranteed  government  operating  cost  with 
contractor  liabilities  should  these  costs  be  exceeded. 

COMPETITION 

o  Contract  provisions  for  each  team  member  to  become  a  viable  compet¬ 
itor  in  production. 

o  Contract  provisions  to  expand  vendor  base  and  maximize  use  of 
socioeconomic  sources  for  parts. 

o  Contract  provisions  to  establish  and  maintain  at  least  two  sources 
for  replenishment  parts. 

PRODUCT  PERFORMANCE 

o  Guaranteed  product  reliability  and  performance  with  specific 
requirements  to  be  demonstrated  at  contractor's  risk  prior  to  production. 

o  Demonstration  of  Reliability  and  Maintainability  (RAM)  requirements 
and  logistics  support  prior  to  full  production. 

o  Contractor  liabilities  for  not  meeting  requirements. 

With  the  award  of  these  competitive  contracts,  the  initiatives  implemented 
shift  risk  to  the  contractor  for  control  of  costs  and  for  successful 
performance.  These  new  initiatives  are  an  innovative  change  to  past 
acquisition  strategy.  For  the  first  time,  industry  has  solidly  committed 
to  guaranteed  production  and  support  costs  at  the  outset  of  development, 
and  it  is  hoped  that  the  T800  contracts  can  serve  as  a  model  for  future 
procurements.  Taken  together,  the  initiatives  now  describe  the  Army's  new 
way  of  doing  business.  In  the  early  stages  of  preparing  the  RFP,  they 
translated  into  what  became  known  as  a  performance-oriented  Request  for 
Proposal . 

The  evaluation  process  concentrated  on  five  major  areas  -  Management, 
Production  Competition,  Technical,  Reliability  and  Maintainability/ 
Integrated  Logistics  Support,  and  Cost.  Many  of  the  lessons  learned  in 
the  process  were  common  to  all  areas;  therefore,  specific  items  of 
interest  may  be  found  more  than  one  place  in  this  document.  Some  of  the 
more  significant  topics  are  as  follows. 

o  The  RFP  was  thoroughly  coordinated  with  potential  offerors  and 
government  agencies  from  the  start.  There  were  no  surprises  in  defining 
the  new  way  of  doing  business;  however,  it  was  essential  that  release  of 
the  RFP  be  preceded  by  pre-proposal  conferences  to  ensure  that  the  indus¬ 
try  was  provided  every  opportunity  to  capture  the  spirit  and  intent  of  the 
government's  requirements  and  objectives. 

o  The  performance-oriented  RFP,  although  new  and  considerably  short¬ 
ened  from  previous  practices,  was  embraced  by  the  offerors  as  a  welcomed 
change  and  an  opportunity  to  produce  against  well-defined  requirements  for 
the  end  product. 


o  Preparations  for  evaluating  future  proposals  will  require  a  new 
look  to  ensure  that  members  of  the  SSEB  are  well  schooled  beforehand. 
Consideration  should  be  given  to  convening  the  board  earlier  -  the  evalua¬ 
tion  team  met  all  schedules;  however,  the  requirement  to  learn  the  new  way 
of  doing  business  should  have  begun  the  week  before  proposals  were 
received . 

o  The  system  involving  inquiries  (Errors,  Omissions,  and  Clarifica¬ 
tions)  on  the  offerors'  proposals  was  extremely  important  to  the  under¬ 
standing  of  their  submissions,  but  the  process  should  have  been  shortened 
to  less  than  3-weeks.  Better  progress  was  made  later  during  face-to-face 
discussions  with  the  offerors. 

o  Frequent  meetings  with  members  of  the  Source  Selection  Advisory 
Council  (SSAC)  were  a  significant  help  to  the  SSEB.  Separate  sessions  of 
this  type  are  highly  recommended  as  a  means  of  reaching  common  understand¬ 
ing  of  differences  in  the  various  proposals  and  uncovering  issues  that 
will  need  special  attention  when  comparing  details  against  the  RFP. 

o  Formal  SSAC  meetings  during  the  evaluation  were  essential.  This 
SSEB  was  involved  in  four  such  meetings,  roughly  a  month  apart.  The 
timing  was  appropriate,  and  each  meeting  forced  the  Board  to  address  open 
issues  without  delay,  requiring  a  dedicated  level  of  self-discipline  to 
reach  specified  targets  on  extremely  tight  (but  workable)  schedules.  It 
is  recommended  that  a  similar  approach  be  used  in  the  future  on  the  basis 
that  it  will  maximi2e  overall  effectiveness  and  avoid  slippage. 

o  Model  contracts  prepared  by  both  parties  and  plans  generated  by  the 
offeror  in  parallel  with  discussions  and  negotiations  provided  an  excel¬ 
lent  baseline  for  the  Best  and  Final  Offer  (BAFO) .  Because  of  tight 
schedules,  this  Board  could  not  provide  the  offerors  with  more  than 
seven  days  to  respond  after  being  officially  notified  on  the  due  date  for 
BAFO.  It  is  recommended  that  the  response  time  be  expanded  to  two  weeks 
and  that  model  contracts  be  provided  to  the  offerors  with  the  notification 
to  submit  the  BAFO. 

This  SSEB  tailored  its  structure  toward  the  new  initiatives,  and  it  became 
clear  early-on  that  special  skills  needed  to  evaluate  the  new  initiatives 
in  RAM/ILS  were  not  readily  available.  As  shown  in  Figure  3,  RAM/ILS  was 
elevated  in  importance  and  became  a  major  area  for  evaluation  for  the 
first  time  in  this  RFP.  Special  needs  for  dealing  with  RAM/ILS  and  its 
elements  (including  MANPRINT)  are  detailed  in  Section  IV. 


As  required  with  all  SSEB  activities,  this  Board  documented  its  findings 
at  the  conclusion  of  the  evaluation.  The  final  report  is  in  four  volumes 
which  have  been  stored  and  secured  by  the  Contracting  Officer.  Volume  I 
contains  the  comparative  analysis  of  proposals  and  the  SSEB's  official 
documents.  Volumes  II,  III,  and  IV  contain  the  Board's  separate  analysis 
of  each  proposal  (one  for  each  offeror's  proposal).  Together,  they 
represent  the  Board's  findings  by  detailed  reference  to  subfactors, 
factors,  elements,  and  areas  of  the  evaluation  structure.  Narratives  are 
provided  and  scores  are  rolled  from  the  lowest  level  through  the  highest 
in  each  area. 


'SWVWm 


The  number  of  key  personnel  assigned  to  form  the  core  of  the  evaluation 
team  was  about  65,  including  the  Chairman,  Deputy,  Area  and  Element 
Chiefs,  Contracting/Legal,  and  administrative/clerical  staffs.  This  total 
is  believed  to  be  far  lower  than  for  previous  SSEBs ;  however,  it  must  be 
noted  that  about  85  short-time  and  part-time  specialists  also  assisted  on 
an  as-needed  basis  (usually  2  weeks  or  less).  This  assistance  was  provid¬ 
ed  by  other  Army  organizations  and  NASA,  as  well  as  elements  of  AVSCOM. 

The  overall  teamwork  and  rapport  developed  throughout  the  process  was 
excellent.  Each  participant  learned  his/her  duties  quickly,  and  the 
leadership  of  this  SSEB  wishes  to  acknowledge  the  success-oriented  spirit 
with  which  the  team  responded  in  a  difficult,  new  approach  to  acquisition. 


SECTION  I 
SSEB  PREPARATIONS 


This  section  of  the  report  addresses  preliminaries  required  to  conduct 
activities  of  the  Source  Selection  Evaluation  Board  (SSEB).  There  were 
many  lessons  learned,  .some  of  which  could  have  improved  efficiency  of  the 
Board  during  the  evaluation.  Because  the  Request  for  Proposal  was  the 
first  of  its  kind,  it  was  necessary  to  adopt  new  disciplines  for  dealing 
with  the  Army's  new  way  of  doing  business.  These  disciplines  demanded 
that  some  key  evaluators  be  trained  to  handle  the  changes  from  past 
practices,  and  most  of  this  training  could  not  be  accomplished  until 
proposals  were  received. 

Flexibility  and  rapid  understanding  were  essential  ingredients  for  both 
the  evaluator  and  the  offeror.  Some  Board  Operating  Procedures  (BOPs) 
from  previous  SSEBs  served  as  a  baseline,  but  changes  needed  to  be  consid¬ 
ered,  particularly  as  related  to  the  Errors,  Omissions,  and  Clarifications 
(EOCs)  process.  The  mechanics  of  the  evaluation  itself  also  uncovered 
needed  improvements.  The  start-up  would  have  been  smoother  if  samples  of 
evaluation  sheets  had  been  developed  and  made  available  for  the  evaluators 
beforehand.  In  addition,  the  on-site  briefing  to  familiarize  the  members 
of  the  SSEB  with  the  task  ahead  cannot  be  overlooked.  It  was  regarded  as 
an  extremely  important  orientation  session  to  expose  evaluators  to  changes 
from  tradition  that  they  could  be  expected  to  face  in  assessing  offerors' 
proposals.  Each  of  these  topics  is  broken  out  and  will  be  addressed  in 
more  depth  in  the  discussions  that  follow. 

A.  REQUEST  FOR  PROPOSAL 

The  Request  for  Proposal  was  structured  to  reflect  the  government’s 
interest  in  a  performance-oriented  product.  Therefore,  the  RFP  stated 
requirements  and  defined  boundaries  without  detailing  methods  by  which  to 
achieve  the  result.  In  essence,  the  typical  "how-to-do-it"  in  previous 
RFPs  was  eliminated,  and  it  was  left  to  the  offeror  to  determine  how  best 
to  meet  requirements.  As  such,  the  RFP  was  much  shorter,  and  the  primary 
section  for  describing  the  product  was  the  System  Specification;  it  also 
was  considerably  reduced  in  size  from  previous  practices.  The  RFP,  with 
this  specification,  gave  the  offeror  flexibility  and  an  opportunity  to 
trade-off  parameters  so  as  to  encourage  optimization.  It  also  called  for 
definitive  plans  to  accomplish  the  tasks.  Therefore,  each  offeror's 
proposal  detailed  his  overall  program  commitments  and  guarantees,  citing 
liabilities  that  he  was  willing  to  accept  for  not  meeting  contractual 
requirements.  That  understanding  between  the  government  and  the  potential 
contractors  was  well  articulated  and  coordinated  in  draft  RFPs  and  prelim¬ 
inary  meetings.  Without  these  reviews,  it  would  have  been  far  more 
difficult  to  instill  the  concepts  of  the  new  way  of  doing  business  into 
the  industry.  It  cannot  be  overemphasized  that  these  pre-RFP  steps  were 
essential  and  that  future  similar  procurements,  likewise,  must  prepare  the 
offerors  in  advance.  As  with  this  procurement,  the  RFP  itself  should 
clearly  show  industry  the  weighting  of  various  areas  of  their  proposal 
(Figure  3).  There  will  be  no  better  way  to  communicate  the  government's 


intentions  or  the  importance  of  each  subject  to  be  addressed  by  the 
evaluators.  Other  RFP-related  lessons  uncovered  during  the  evaluation 
process  are  discussed  below. 
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1.  CONTRACTUAL  PLANS  -  The  attempt  to  eliminate  the  "how-to-do-it” 
from  the  RFP  went  far  toward  simplifying  the  government's  requirements. 

In  the  future,  it  would  best  serve  the  interests  of  the  government  and  the 
offeror  to  state  whether  the  submitted  plans  are  to  become  a  part  of  the 
contract  or  are  to  be  submitted  for  evaluation  purposes  only.  Included  in 
this  category  are  the  numerous  test/development  plans,  competition  plans, 
and  RAM/ILS  plans. 
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2.  OFFEROR’S  MANAGEMENT  -  During  the  evaluation  process,  it  became 
clear  that  the  Management  Area  of  the  evaluation  would  have  been  better 
handled  under  the  other  four  Areas  (see  Figure  3)  -  Production 
Competition,  Technical,  RAM/ILS,  and  Cost.  Because  of  the  desire  to  place 
more  control  of  the  development  program  into  the  contractor's  hands  than 
in  the  past,  the  basic  understanding  of  management  aspects  became  closely 
associated  with  the  program  plans  that  were  proposed  under  the  four  main 
areas  noted  here.  In  addition,  the  initiatives  encouraged  by  the  new  way 
of  doing  business  led  government  evaluators  assigned  to  the  Management 
Area  to  overlap  the  others,  and  special  care  was  exercised  to  prevent 
evaluating  the  same  topic  in  two  or  more  places.  This  SSEB  is  certain 
that  no  such  conflicts  in  the  evaluation  plan  occurred;  however,  the  team 
remained  alert  to  that  potential.  Concerns  were  resolved  by  sharing 
findings  and  completing  the  evaluation  in  the  most  appropriate  area. 

Future  RFPs ,  therefore,  should  include  management  elements  or  factors  in 
other  areas  where  they  more  properly  apply  (not  as  a  separate  area),  but 
even  under  such  a  breakout,  caution  must  be  applied  to  ensure  that 
cross-overs  do  not  lead  to  evaluating  a  specific  management  topic  in  more 
than  one  place. 


3.  CONTRACT  DATA  REQUIREMENTS  LISTS  -  In  keeping  with  the  desire  to 
eliminate  complexity  in  the  RFP  and  in  performance  of  a  follow-on  con¬ 
tract,  Contract  Data  Requirements  List  (CDRLs)  were  reviewed  to  establish 
whether  they  truly  represented  government  needs.  Of  the  123  reviewed,  65 
were  discarded.  This  measure  to  streamline  the  RFP  and  define  the  real 
requirements,  at  first,  was  met  with  resistance;  however,  the  result  was  a 
reduced  burden  on  the  government  and  the  offeror,  alike.  As  became 
evident  through  the  later  evaluation,  caution  must  be  exercised  to  avoid 
eliminating  the  government's  tools  for  measuring  contractor  performance. 
This  concern  involves  CDRLs  on  parts  lists,  manuals,  and  test  plans,  which 
initially  were  dropped,  but  later  added  in  modified  form. 

4.  OVERALL  SCHEDULE  -  The  entire  process  from  preparation  of  the  RFP 
to  completion  of  the  SSEB  reports  can  take  a  year  or  more,  depending  on 
the  complexity  of  the  acquisition  strategy  employed.  For  this  SSEB,  the 
RFP  was  started  in  June  1984  and  released  to  industry  formally  on  5 
December  1984.  The  activity  was  closed  out  on  30  August  1985,  with 
completion  of  all  documentation  requirements.  The  elapsed  time,  14 
months,  proved  to  be  adequate,  but  it  required  some  unusual  steps,  such  as 
parallel  fact-finding  discussions  and  negotiations  with  three  offerors, 
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all  meeting  in  different  buildings  on  the  same  days.  This  overlapping 
should  be  avoided  if  at  all  possible.  For  the  T800,  the  situation  result¬ 
ed  from  a  learning  process  involving  a  need  to  conduct  lengthy  and  de¬ 
tailed  discussions  on  contractual  plans,  guarantees,  and  numerous 
commitments  that  reflected  the  Army's  new  way  of  doing  business.  This 
learning,  although  difficult  and  tedious  for  the  offerors,  as  well  as  the 
evaluation  team,  was  essential  to  the  selection  of  winners.  Details  of 
this  Board's  schedule  are  provided  in  Figure  4.  A  suggested  schedule  for 
planning  similar  future  SSEB  activities  is  provided  in  Figure  5.  Both 
schedules  will  be  discussed  in  more  detail  in  Section  II  of  this  report. 

B.  SSEB  STAFFING 

In  staffing  an  SSEB,  it  is  believed  essential  that  the  Area  Chiefs, 
Contracting  Officer,  Administrative  Officer,  and  key  Element  Chiefs  be 
assigned  early  enough  to  play  a  significant  role  in  writing  the  RFP.  They 
should  be  in  place  on  a  full-time  basis  with  adequate  office  space  to 
operate  as  a  team.  Likewise,  the  Chairman  and/or  the  Deputy  Chairman 
should  devote  full  energies  to  the  preparation  of  the  RFP  well  in  advance 
of  its  formal  release  to  industry.  It  is  suggested  that,  as  a  minimum, 
the  team  leaders  (Chairman,  Deputy,  and  Chiefs)  be  named  six  months  prior 
to  the  scheduled  issue  date  of  the  RFP  and  that  the  Chairman  and/or  Deputy 
be  prepared  for  full-time  service  2  months  before  issue.  By  so  doing,  the 
same  individuals  preparing  the  RFP  will  be  available  to  develop  the  Source 
Selection  Evaluation  Plan,  identify  needs  for  Board  Operating  Procedures, 
and  define  staffing  requirements  for  the  entire  SSEB.  The  carry  over  from 
one  to  the  other,  clearly,  will  enhance  the  board's  effectiveness,  espe¬ 
cially  in  the  early  days  of  the  formal  evaluation  of  proposals.  Relative 
to  the  Army's  new  way  of  doing  business,  the  importance  of  early  involve¬ 
ment  is  further  emphasized,  and  key  participants  must  be  well  versed  in 
all  details  of  the  acquisition  strategy,  new  initiatives,  and  long-range 
objectives . 

The  task  of  identifying  specific  skills  needed  for  the  evaluation  and  the 
selection  of  the  remainder  of  the  board  must  begin  immediately  after 
release  of  the  RFP.  Each  key  participant  in  writing  the  RFP  and  the 
Evaluation  Plan  should  play  a  role  based  on  intimate  knowledge  of  the 
requirements.  These  needs  must  be  spelled  out  in  sufficient  detail  to 
permit  an  efficient  search  for  the  needed  skills.  Without  question,  the 
tendency  will  be  to  name  specific  individuals;  however,  such  requests 
should  be  avoided  unless  they  are  provided  in  the  context  of  requesting  a 
desired  level  of  skill  and  talent.  Working  with  the  administrative  staff, 
the  Chairman  or  Deputy  must  develop  a  potential  roster  by  seeking  the 
assignments  formally  through  appropriate  channels,  and  these  commitments 
to  the  SSEB  will  be  extremely  difficult  to  secure.  The  task  will  become 
the  most  time-consuming  effort  and  will  involve  assignments  to  Contract¬ 
ing,  Legal  and  Administration,  along  with  the  evaluation  areas.  It  will 
fall  upon  the  Chairman  or  Deputy  to  follow-up  persistently  with  inquiries, 
some  of  which  will  involve  requests  for  assistance  from  the  other  servic¬ 
es.  This  process  may  well  involve  high-level  exposure  to  emphasize  the 
framework,  acquisition  strategy,  and  importance  placed  on  the  procurement 
at  hand.  Every  effort  must  be  made  to  establish  the  roster  clearly  at 


least  30  days  before  proposals  are  received,  and  assignment  letters  should 
follow  immediately  thereafter. 
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To  minimize  difficulties  in  assignment  of  evaluators,  action  should  be 
taken  to  develop  and  implement  AVSCOM  and/or  AMC  policy  at  the 
director/office  chief  level  to  provide  the  appropriate  personnel.  Consid¬ 
ering  the  importance  of  such  acquisitions,  needed  personnel  should  be 
provided  willingly.  Among  the  critical  skills  most  sought,  participants 
must  demonstrate  good  oral  and  written  communication,  as  well  as  analyti¬ 
cal  strengths.  In  some  areas  such  as  RAM/ILS,  consideration  should  be 
given  to  training  and  development  of  individual  skills  in  such  programs  as 
LOGAMP . 


In  addition  to  these  assignments,  it  was  extremely  valuable  to  this  SSEB 
that  a  highly  qualified  individual  be  available  on  a  full-time  basis  to 
interface  with  all  areas  and  functional  units  of  the  Board.  His  purpose 
was  to  coordinate  all  details  of  formal  presentations  to  the  SSAC  and  SSA, 
while  also  acting  as  the  focal  point  for  orchestrating  numerous  other 
tasks  conducted  jointly  among  the  Board  leadership.  This  assessment  was 
significant  in  permitting  the  Board  to  stay  on  track  and  proved  indis¬ 
pensable  in  maintaining  uniformity  of  all  data  presented. 


The  tracking  of  EOCs,  compilation  of  negotiation  issues,  and  the  taking 
and  preparation  of  required  meeting  minutes  were  time  consuming  and 
burdensome  for  both  element  and  factor  chiefs.  Future  SSEBs  should  staff 
each  area  (or  even  element)  with  dedicated  people  to  handle  these  require¬ 
ments.  Junior  personnel  (interns  and  trainees)  would  be  good  candidates 
for  these  positions,  and  the  experience  gained  would  be  invaluable  in 
developing  a  cadre  of  experienced  people  for  future  boards. 


As  a  part  of  this  staffing  requirement,  it  should  be  noted  that  special¬ 
ized  equipment  (computers,  word  processors)  needed  by  the  board  must  be 
available  early  and  that  training  for  their  use  must  begin  as  soon  as 
practicable  to  permit  smooth  transition  into  the  board  activity. 


BOARD  OPERATING  PROCEDURES 


Board  Operating  Procedures  (BOPs)  are  an  established  requirement  to 
provide  guidance  to  personnel  assigned  to  each  Source  Selection  Evaluation 
Board  (SSEB).  In  the  list  that  follows  it  may  be  seen  that  the  operation 
of  this  Board  involved  17  such  BOPs,  most  of  which  are  widely  accepted  for 
use  on  other  similar  activities.  For  future  boards,  it  may  be  advanta¬ 
geous  to  consider  making  some  of  these  BOPs  available  to  the  offerors  in 
that  they  may  prove  helpful  to  understanding  administrative  details. 

Those  that  are  suggested  for  this  purpose  are  BOP  numbers  2,  4,  12,  and 
13. 


1.  Personnel  In-processing  Procedure  -  Provide  guidance  for  process¬ 
ing  of  incoming  personnel  for  admittance  to  the  SSEB  Area. 


2.  Security  Procedures  -  Define  requirements  for  control  of  classi¬ 
fied  or  sensitive  information  provided  to  or  emanating  from  the  SSEB. 
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3.  Standards  of  Conduct  -  Provide  guidance  for  personal  and  profes¬ 
sional  conduct  relative  to  SSEB  matters. 

4.  Parking  -  Provide  guidance  for  the  parking  of  privately  owned 
conveyance  of  personnel  assigned  to  and/or  on  official  business  with  the 
SSEB. 

5.  Use  of  Telephones  -  Provide  guidance  for  telephone  usage  and 
procedures,  including  restrictions. 

6.  Hours  of  Duty  -  Define  normal  duty  hours  and  procedures  for 
overtime . 

7.  Time  and  Attendance  Reporting  -  Establish  uniform  method  of 
recording  normal  hours  worked,  overtime  worked,  sick  and  annual  leave 
taken. 

8.  Claim  for  Reimbursement  for  Expenditures  on  Official  Business  - 

Define  procedures  for  processing  Claim  for  Reimbursement  for  Expenditures 
on  Official  Business  and  Nonavailability  of  Official  Transportation  for 
Local  Travel,  Privately-owned  Vehicle  Authorization. 

9.  Control  of  Reproduction  -  Define  responsibilities  for  control  of 
reproduction  of  documents. 

10.  Typing  Pool  Procedures  -  Establish  a  uniform  method  for  process¬ 
ing  and  controlling  typed  material  emanating  from  the  source  selection 
evaluation. 

11.  Offerors'  Requests  for  RFP  Clarification  -  Establish  a  uniform 
method  of  responding  to  offerors'  questions  to  the  government. 

12.  EOC  Reporting  -  Establish  a  uniform  method  of  reporting  errors  or 
omissions  discovered  in  the  offerors'  proposals  and  for  requesting  clari¬ 
fication  when  such  information  is  essential  for  conducting  the  evaluation. 

13.  Amendments  to  Request  for  Proposal  -  Establish  a  uniform  method 
to  amend  the  RFP,  subsequent  to  original  proposal  submittal  date. 

14.  "Eyes  Only"/Back  Channel  Messages  -  Provide  necessary  guidance 
and  delineate  responsibilities  and  procedures  to  ensure  efficient  process¬ 
ing  of  "Eyes  0nly"/Back  Channel  Messages  for  the  Chairman,  SSEB. 

15.  Lessons  Learned  Reports  -  Establish  control  for  preparation  of 
lessons  learned  reports. 

16.  Individual  Performance  Rating  Scheme  -  Was  not  published  for  this 
SSEB,  but  should  have  been  provided  with  samples  of  performance  standards, 
performance  appraisals  and  guidance  as  to  the  rating  scheme  of  both 
military  and  civilian  personnel  assigned  for  a  period  90  and  120  days, 
respectively.  Without  it,  supervisors  used  methods  established  by  their 
parent  organizations. 


17.  Departure  Clearance  Procedures  -  Provide  guidance  for  processing 

personnel  departing  the  SSEB. 

The  following  is  a  list  of  Board  Operating  Procedures  that  would  have  been 
helpful  had  they  been  established  prior  to  the  beginning  of  the  SSEB. 

1.  Procedures  for  submitting  partial  pay  vouchers  for  personnel 
assigned  to  the  SSEB  who  are  TDY. 

2.  Procedures  for  typing  TDY  orders  for  assigned  personnel  required 
travel  in  support  of  the  SSEB. 

3.  Typing  guide  for  all  clerical  personnel,  to  include  signature 
blocks  for  Chairman,  Deputy,  and  Area  Chiefs. 

A.  Instructions  for  the  preparation  of  the  final  report  (for  the 
typist  and  the  evaluators). 

D.  EVALUATION  PLAN 


The  Source  Selection  Evaluation  Plan  addresses  responsibility  for  source 
selection  as  related  to  the  Source  Selection  Authority  (SSA)  and  the 
Source  Selection  Advisory  Council  (SSAC) ,  in  conjunction  with  the  Source 
Selection  Evaluation  Board  (SSEB).  Preparation  was  in  accordance  with 
guidance  set  forth  in  Army  Regulation  715-6,  September  1970,  "Proposal 
Evaluation  and  Source  Selection";  USAMC  Supplement  1  to  AR  715-6, 

3  November  1970;  DARCOM  Pamphlet  715-3,  October  1980;  and  Subpart  15.6 
(Source  Selection)  of  the  Federal  Acquisition  Regulation  (FAR).  The 
Evaluation  Plan  reflects  the  T800  engine  program  requirements,  defines 
principal  responsibilities,  describes  the  evaluation  process,  and  provides 
a  description  of  the  administrative  aspects  related  to  the  operation  of 
the  SSEB.  While  the  purpose  of  the  Evaluation  Plan  is  to  assure  impar¬ 
tial,  equitable,  and  comprehensive  evaluation  of  competitive  proposals,  it 
must  lead  to  selection  of  a  contractor  whose  proposed  program  will  provide 
the  optimum  approach  and  best  value  to  the  government.  Therefore,  it  must 
describe  detailed  plans  and  procedures  for  the  SSEB,  establish  the  struc¬ 
ture  of  an  administrative  system  for  evaluating  and  scoring  proposals,  and 
document  the  baseline  for  recording  and  supporting  the  SSEB. 


The  Evaluation  Plan  and  the  Request  for  Proposal  (RFP)  must  be  closely 
linked  documents.  For  the  T800 ,  the  Evaluation  Plan  was  prepared  in 
parallel  with  the  RFP.  Evaluation  criteria,  as  reflected  in  the  Plan, 
were  based  on  requirements  shown  in  Section  C  and  the  System  Specif ication 
of  the  RFP,  and  the  evaluation  approach  was  mirrored  in  Section  M  of  the 
RFP,  Evaluation  Factors  for  Award.  It  was  deemed  imperative  that  the 
offerors  be  provided  insights  to  be  used  by  the  SSEB  in  evaluating  propos¬ 
als  and  that  precise  language  be  used  to  communicate  the  importance  of 
each  of  the  evaluation  areas.  One  of  the  innovations  of  this  RFP  was  that 
the  distribution  of  weights  assigned  to  areas  of  evaluation  (down  to  the 
factor  level)  was  provided  in  detail  (Figure  3)  so  that  the  relative 
importance  of  these  topics  couli  be  understood  easily. 
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Preparation  of  the  Evaluation  Plan  was  iterative  in  nature.  As  RFP 
requirements  changed,  so  did  the  Plan  and  its  attendant  evaluation 
criteria.  These  changes  were  a  result  of  preliminary  reviews  (prior  to 
formal  release),  wherein  drafts  of  the  RFP  were  provided  to  industry  and 
to  other  government  agencies  for  review  and  comment.  Throughout,  the 
objective  was  to  delete  "how-to's”  in  an  attempt  to  make  the  RFP  as 
performance  oriented  as  possible.  Each  step,  including  formal  briefings, 
was  iterated  with  the  SSAC  and  resulted  in  refinements  of  the 
previously-mentioned  weighting,  scoring,  and  definitions. 


A  large  portion  of  the  Evaluation  Plan  was  devoted  to  Production  Competi¬ 
tion,  RAM/ILS  and  Cost.  All  three  represented  firsts  -  Production  Compe¬ 
tition  became  a  separate  area  with  required  commitments  up  front  in  the 
program;  RAM/ILS  became  a  separate  area,  weighted  and  scored;  and  Cost 
became  a  weighted  and  scored  area.  To  reflect  the  results  of  these 
thoroughly  reviewed  changes  in  practice,  final  iterations  in  an 
eleventh-hour  effort  were  required  to  provide  detailed  criteria 
(.Appendix  II  to  the  Evaluation  Plan).  In  essence,  these  criteria  were  the 
instructions  to  the  evaluators. 

It  follows  from  the  above  that  the  iterative  process  (including  require¬ 
ments  and  guidance)  must  be  concluded  early  and  that  the  importance  of 
developing  and  coordinating  detailed  evaluation  criteria  cannot  be  over¬ 
stated.  Criteria  must  be  written  clearly,  and  there  must  be  no  ambiguity. 
Therefore,  it  is  essential  that  the  authors  of  the  RFP  be  available  to 
translate  the  iterations  into  understandable  criteria  for  the  Evaluation 
Plan. 
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E.  ON-SITE  BRIEFING 

For  most  of  the  evaluators  and  administrative  personnel  assigned  to  the 
SSEB,  the  on-site  briefing  was  their  first  exposure  to  the  task  at  hand. 

It  also  was  the  first  opportunity  for  many  to  develop  the  essential  team 
relationship-  For  those  reasons,  the  on-site  briefing  for  the  T800  SSEB 
was  held  before  proposals  were  received.  For  any  board,  it  should  be  a 
requirement  to  cover  background  and  preparation  of  the  RFP,  the  Evaluation 
Plan,  and  rules  of  conduct.  The  T800  briefing  took  place  one  day  in 
advance,  and  special  efforts  were  made  to  describe  the  new  way  of  doing 
business.  Emphasis  was  placed  on  the  initiatives  of  the  RFP  and  the  steps 
taken  to  encourage  offerors  to  present  innovations.  Although  this  orien¬ 
tation  served  the  purpose  well,  it  is  clear  that  the  team  needed  two  more 
days  of  exchange  with  the  leadership  of  the  SSEB  to  capture  the  approach 
before  beginning  the  evaluation.  An  outline  of  the  presentation  used  for 
the  T800  on-site  briefing  is  as  follows 

o  Purpose 

o  Responsibilities 

o  Engine  Requirements 


SECTION  II 


EVALUATION  OF  PROPOSALS 


The  process  of  evaluating  proposals  involves  a  number  of  routines  for 
which  guidance  is  provided  in  Army  Regulation  No.  715-6,  Federal  Acquisi¬ 
tion  Regulation  15.6,  and  DARCOM  Pamphlet  No.  715-3.*  With  these  guide¬ 
lines  and  the  plans  described  in  Section  I  of  this  after-action  report, 
the  SSEB  was  prepared  to  address  proposals.  It  is  emphasized  that  the 
highest  standards  of  uniformity  must  be  exercised  by  any  board,  and  it  is 
essential  that  there  be  precise  adherence  to  this  standard  by  all  partici¬ 
pants  in  all  data  developed.  Each  step  identified  in  the  paragraphs  of 
this  section  (below)  required  detailed  visibility  and  control  at  the  area 
level;  however,  it  must  be  the  responsibility  of  the  Chairman/Deputy  to 
establish  the  methodology  that  will  meet  the  standard  of  uniformity  and 
integrity . 

A.  INITIAL  REVIEW 

Evaluators  and  chiefs,  alike,  should  be  permitted  sufficient  time  to 
review  proposal  data  and  become  familiar  with  an  offeror's  approach  to 
meeting  the  RFP  requirements.  An  overview  briefing  by  the  offeror  is  a 
most  welcome  introduction  to  acquaint  the  board  with  highlights  and 
features  of  each  proposal.  Depending  on  the  number  of  proposals  received, 
it  will  be  most  appropriate  to  complete  these  sessions  in  one  day  with 
each  offeror  given  about  an  hour  to  discuss  his  response  to  the  RFP.  In 
addition,  these  presentations  should  take  place  within  a  day  or  two  after 
proposals  are  received,  and  they  should  be  directed  at  the  content  of  the 
proposal  volumes  and  business  arrangements.  For  the  T800  Board,  the 
presentations  included  about  an  hour  and  a  half  of  discussion  and  video 
tape.  Although  these  tapes  were  useful,  the  offerors  could  have  presented 
the  same  information  more  comfortably  and  less  formally  with  vu-graph 
slides  and  with  a  reduced  burden  during  a  time  when  critical  proposal 
delivery  schedules  had  highest  priority.  It  is  the  Board's  opinion  that 
the  offeror  would  have  preferred  to  concentrate  on  final  proposal  details 
and  follow-up  with  a  short,  executive  summary  video  tape,  perhaps  a  week 
after  due  date  for  the  proposal.  Despite  this  belief,  each  offeror  met 
the  schedule  and  provided  excellent  initial  reviews.  Each  of  them  also 
believed  that  the  tapes  served  their  own  needs  well,  in  addition  to 
meeting  the  government's  desires  for  a  Concise  summary. 

For  the  first  week  to  ten  days  of  the  initial  review,  the  SSEB  should 
concentrate  on  understanding  each  proposal,  so  as  to  permit  development  of 
the  basis  for  inquiries  in  the  EOC  process  (requests  for  information  on 
Errors,  Omissions,  and  Clarifications).  It  is  recommended  that  evaluation 


It  is  suggested  that  each  of  these  documents  be  made  available  to 
offerors  when  an  RFP  is  released.  The  insights  gained  from  these  refer¬ 
ences  will  prove  helpful  to  the  offeror  by  enhancing  understanding  of  the 
government's  proposal  evaluation  process. 
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of  proposal  details  not  begin  until  this  baseline  has  been  set  and  all 
area  chiefs  have  had  ample  time  to  coordinate  their  approaches  and  commu¬ 
nicate  their  concerns.  As  in  the  past,  many  questions  will  arise,  and  a 
large  share  of  them  can  be  resolved  early-on  with  internal  communication 
and  without  requiring  an  EOC .  The  board  members  must  work  together  to 
avoid  duplication  and  turmoil  for  there  can  be  no  substitute  for  thorough¬ 
ly  reading  the  proposals  and  insisting  on  detailed  coordination. 


In  establishing  a  starting  point  for  the  evaluation,  each  area  chief 
should  define  a  due  date  for  an  initial  finding  at  the  lowest  level  in  the 
evaluation  structure.  These  findings,  although  preliminary,  will  surface 
the  concerns  within  that  area  and  will  lend  substance  to  the  EOCs .  During 
these  first  weeks,  daily  staff  meetings  with  the  SSEB  leadership  are 
recommended  as  the  means  to  communicate  details  of  planned  EOCs  and  to 
foster  interfaces  among  evaluators  in  all  areas.  Because  the  benefits 
gained  by  sharing  information  between  areas  on  designated  subjects  is 
vital,  the  process  must  involve  at  least  the  area  chiefs,  and  it  must  not 
be  left  to  take  place  without  guidance.  Further  discussion  of  the  EOC 
process  is  provided  in  paragraph  B,  which  follows. 

B.  ERRORS,  OMISSIONS,  AND  CLARIFICATIONS 

A  detailed  discussion  of  this  process  can  be  found  in  DARCOM-P  715-3.  In 
brief  it  states,  "In  reviewing  a  contractor's  proposal  prior  to  scoring 
it,  evaluators  of  the  SSEB  may  encounter  errors,  omissions,  and  deficien¬ 
cies  in  varying  degrees  of  substance.  Some  are  superficial  and  easily 
correctable  while  others  are  more  serious."  For  the  T800  SSEB,  a  formal 
Error,  Omission,  and  Clarification  (EOC)  inquiry  was  used  to  gather  enough 
initial  data  to  establish  a  baseline  for  the  original  proposal. 

The  process  was  indispensable  to  enhancing  initial  understanding  and 
establishing  a  baseline  for  follow-on  efforts.  However,  it  was  clear  that 
this  tool  could  easily  be  used  to  collect  data  that  was  either  superfluous 
or  could  be  better  handled  during  the  follow-on,  face-to-face  discussions. 
The  sole  objective  of  the  EOC  process  must  be  to  permit  the  collection  of 
information  needed  to  understand  the  proposal.  The  process  should  be 
short  (no  more  than  two  weeks)  and  cover  a  well-defined  period  with  a 
specific  start  and  end  date.  It  must  not  be  permitted  to  enlarge  and 
overtake  other,  more  valuable  fact-finding  responsibilities,  such  as 
direct  discussions  with  the  offerors.  These  face-to-face  discussions; 
therefore,  should  begin  as  soon  as  possible  after  the  initial  evaluation 
so  as  to  ensure  that  the  government's  positions  and  concerns  are  adequate¬ 
ly  conveyed  to  the  offerors.  Based  on  the  innovations  sought  in  the  RFP, 
the  discussions,  although  sometimes  repetitive,  were  indispensable  to  the 
success  of  the  program,  more  so  than  the  EOC  process. 

During  the  evaluation,  offerors  were  quick  to  note  that  they  regarded  the 
EOC  as  far  less  significant  to  the  process  and  were  pleased  at  the  oppor¬ 
tunity  to  meet  with  the  Board.  Each  offeror  also  recommended  that  the  EOC 
process  be  kept  to  a  minimum  and  that  discussions  and  follow-on  negotia¬ 
tions  begin  as  soon  as  possible. 


This  program  experience  emphasized  the  requirement  for  a  pre-evaluation 
EOC  orientation,  augmented  with  examples  of  type,  coordination,  staffing 
and  tracking.  In  accordance  with  DARCOM-P  715-3,  coordination  between  the 
evaluation  areas  should  be  sufficient  to  assure  clear  and  precise  state¬ 
ments  of  their  particular  requirements  on  a  specific  request.  For  the 
T800  SSEB,  daily  staff  meetings  were  necessary  for  coordination,  but,  even 
then,  some  detailed  concerns  could  not  be  avoided.  Although  the  computer 
system  used  for  tracking  by  this  board  was  available  to  most  Areas  and  was 
a  good  tool,  the  focal  point  for  control  and  tracking  of  all  EOC  actions 
(assignment  of  board  control  numbers,  quantities,  and  format  assembling 
and  inputting  into  the  system)  should  be  with  the  Contracting  Officer.  In 
addition,  EOCs  and  responses  should  be  filed  by  areas,  as  well  as  by 
control  numbers.  Each  SSEB  must  make  all  administrative  arrangements  and 
assignments  clearly,  well  in  advance,  to  ensure  that  the  learning  process, 
including  internal  communications  with  all  Areas,  are  handled  effectively. 

However  future  SSEBs  are  run,  it  should  be  recognized  that  formal,  written 
inquires  are  time  consuming  and  not  always  satisfactory  in  their  response. 
Face-to-face  discussions  are  less  manpower  intensive  and  enable  a  great 
deal  of  information  to  be  passed  with  greater  assurance  of  understanding. 

C.  FACT-FINDING  DISCUSSIONS 

Although  the  EOC  process  proved  to  be  very  beneficial  in  the  initial  phase 
of  the  SSEB  (relative  to  errors  and  omissions),  it  was  of  limited  value 
fc-r  clarifications.  Questions  and  answers  often  were  difficult  to  commu¬ 
nicate  in  writing  via  the  EOC  process,  due  primarily  to  the  complex  nature 
of  the  proposal  submitted  in  response  to  the  performance-oriented  RFP. 

The  EOC  process,  however,  did  complement  the  subsequent  fact-finding 
discussions  and  served  as  a  basis  for  identifying  deficiencies  and  high 
risks.  They  also  aided  in  the  clarification  and  understanding  of  the 
government’s  requirements  and  philosophies.  On  the  T800  SSEB,  the  EOC 
process  unavoidably  overlapped  those  discussions,  and  there  were  times 
when  a  discussion  topic  had  already  been  submitted  by  the  offeror  in  an 
EOC  but  not  yet  received  by  the  evaluator.  Therefore,  if  the  schedule 
permits,  it  is  recommended  that  on  future  SSEBs,  fact-finding  discussions 
not  begin  until  the  EOC  process  has  been  completed. 

Fact-finding  discussions,  primarily,  were  conducted  by  area  (Manage¬ 
ment,  Production  Competition,  Technical,  RAM/ILS,  and  Cost),  which  permit¬ 
ted  the  offerors  and  government  specialists  to  participate  on  a 
pre-scheduled  basis.  Clearly,  it  was  advantageous  to  both  parties  to  have 
a  pre-planned  agenda  with  open  questions;  therefore,  in  most  cases,  topics 
were  submitted  to  the  offeror  at  least  three  days  in  advance.  To  focus 
the  discussions,  government  contracting  and  legal  specialists  were  re¬ 
quired  to  participate  fully  and  exercise  controls  to  avoid  technical 
leveling  and  transfusion  or  any  other  aspect  that  might  unfavorably  impact 
a  fair  and  competitive  environment.  For  the  T800  SSEB,  discussions  were 
conducted  separately  with  all  offerors,  but  because  of  schedule  constraints 
imposed  for  contract  award,  discussions  often  took  place  on  area  subjects 
in  parallel  sessions  on  the  same  day.  Because  of  innovations  sought  by 
the  government,  the  task  of  understanding  details  was  greater  than  antici¬ 
pated,  and  it  became  necessary  to  add  personnel  to  the  legal  staff  on  a 


temporary  basis  to  cover  several  meetings  in  parallel.  Concurrent  discus¬ 
sions,  obviously,  present  a  problem  in  that  the  strong  interrelationship 
between  areas  can  require  that  key  evaluators  be  available  for  more  than 
one  meeting  at  the  same  time. 

Although  responsibility  for  all  exchanges  with  the  offerors  must  be  with 
the  Contracting  Officer,  it  is  important  that  the  responsibility  for 
taking  minutes  of  discussions  and  negotiations  also  be  clearly  defined. 
Unlike  the  EOC  process,  which  provides  for  an  audit  trail  in  writing, 
minutes  are  the  only  record.  They  must  be  accurate  and  complete,  and,  to 
be  effective,  they  should  be  provided  to  the  offeror  within  3-5  working 
days  following  the  session.  In  that  way,  these  minutes  can  provide  an 
opportunity  to  reflect  the  agenda  items,  state  the  government's  interpre¬ 
tation  of  the  discussions,  and  identify  action  items  and  issues.  To  be 
most  effective,  these  minutes  should  be  prepared  by  the  area  (RAM/ILS,  for 
example)  from  which  the  discussion  topics  were  generated.  In  all  cases, 
however,  review  and  distribution  of  minutes  must  remain  the  responsibility 
of  the  Contracting  Officer. 

D.  NEGOTIATIONS 

Negotiations  with  the  offerors  are  best  pursued  after  tcrP  requirements  and 
proposed  responses  are  clearly  understood  by  all  parties.  Included  in 
this  step  are  complex  discussions  on  assessments  of  trade-offs  and  risk  as 
they  relate  to  the  RFP  and  its  System  Specification.  It  is  an  opportunity 
for  both  parties  to  propose  approaches  to  development  of  contractually 
binding  language. 

For  the  T800  SSEB,  negotiations  were  conducted  over  a  broad  range  of 
topics  with  emphasis  on  organizational  issues,  guarantees,  liabilities, 
long-range  commitments,  and  warranties  in  the  four  major  areas  (Production 
Competition,  Technical,  RAM/ILS,  and  Cost).  In  retrospect,  it  is  diffi¬ 
cult  to  identify  where  fact-finding  ended  and  negotiations  began  -  this 
Board  moved  from  one  to  the  other,  while  emphasizing  requirements,  in  a 
continuous  process  without  pause.  Model  contracts,  which  formed  the 
framework  for  eac’  offeror  to  build  upon,  were  based  on  the  earlier 
discussions  and  these  negotiations,  and  the  contents  are  described  below. 

E .  MODEL  CONTRACTS 

Development  of  model  contracts  proved  to  be  an  extremely  beneficial  fallout 
of  discussions  with  the  offerors.  The  first  model  contract  was  provided 
to  the  offerors  approximately  one  week  prior  to  the  initial  negotiation 
sessions.  These  models  became  the  working  documents  used  by  the  parties 
during  negotiations  and  enhanced  the  proceedings  by  focusing  the  discus¬ 
sions.  In  this  way,  the  discussions  became  highly  structured  and  most 
respons  ive . 

As  intended,  these  working  documents  were  continually  updated  after  each 
session  and  proved  to  be  an  effective  audit  trail  in  the  evolution  of 
final  contracts  submitted  to  the  offerors  with  the  request  for  Best  and 
Final  Offer  ( BAFO ) .  Evolution  of  the  model  contract  was  on  an  alternating 
basis  between  the  parties  -  agreements  reduced  to  writing,  development  of 
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joint  understanding,  open  discussion,  and  resubmittal  at  the  next  session. 
This  check  and  balance  ensured  that  mutual  agreement  was  reached  in  the 
negotiated  concepts  and  that  the  necessary  contractually-binding  language 
represented  commitments  agreeable  to  both  parties. 

Program  plans  were  included,  and  it  was  only  through  this  concentrated 
effort  of  continuous  updating  that  the  offerors  were  in  a  position  to 
respond  to  the  request  for  BAFO  in  only  seven  days. 

F.  BEST  AND  FINAL  OFFER  (BAFO) 

The  request  for  BAFO  was  provided  by  written  notification  through  the 
Contracting  Officer  as  required  by  FAR  and  contained  detailed  information 
and  instructions  relative  to  submission.  Three  copies  of  the  final  model 
contract  were  furnished.  Included  were  special  instructions  to  provide 
(fill  in)  specific  competitive  information,  such  as  prices,  incremental 
funding  requirements  and  schedules.  The  offerors  also  were  requested  to 
submit  two  signed  original  copies,  which  were  used  during  the  BAFO  evalua¬ 
tion  process.  Because  the  government  had  retained  the  right  to  award  more 
than  one  contract  (and  ultimately  awarded  two),  the  offerors  were  in¬ 
structed  to  place  restrictive  markings  on  their  BAFO  to  the  extent  that 
they  believed  were  supportable  in  the  light  of  the  competition  that  likely 
would  continue  through  the  down-select  at  PFR. 

Partly  related  to  this  concern,  the  offerors  were  requested  to  submit  a 
"shell"  contract  which  deleted  all  information  considered  by  them  to  be 
COMPETITION  SENSITIVE  and/or  PROPRIETARY.  This  form  of  protection  evolved 
from  prior  agreement  (between  the  SSEB  and  the  offerors)  that  certain 
contract  data  should  not  be  subject  to  release  via  Freedom  of  Information 
Act  (FOIA)  requests.  However,  because  the  SSEB  did  not  have  authority  to 
protect  such  data,  contracts  would  be  referred  to  higher  level  of  review. 


A  final,  yet  very  significant,  point  in  BAFO  is  that  because  of  late 
fiscal  changes  and  related  concerns  that  can  affect  the  overall  program 
plans,  it  is  recommended  that  any  future  requests  for  BAFO  clearly  define 
a  period  for  which  the  submission  must  remain  valid. 


G.  FINAL  EVALUATION  AND  SCORING 

It  should  be  noted  that  each  step  in  the  evaluation  process  involved 
reaching  some  measure  of  a  preliminary  conclusion;  however,  it  must  be 
emphasized  that  the  potential  contractor's  Best  and  Final  Offer  supersedes 
all  previous  findings.  To  recount  the  steps  to  this  point,  the  Board 
continued  to  enlarge  its  perspectives  through: 


o  initial  reviews  of  proposals; 
o  issuing  inquiries  ( EOC ) ; 


o  holding  fact-finding  discussions; 
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o  conducting  negotiations; 


o  developing  model  contracts;  and 

o  studying  Best  and  Final  Offers  (BAFO). 

Along  the  way,  each  evaluation  area  had  the  opportunity  to  develop  prelim¬ 
inary  scores  for  evaluation  purposes;  however,  each  also  was  instructed 
that  all  such  documentation  up  to  BAFO  was  subject  to  change.  Therefore, 
it  was  required  that  any  narratives  and  scoring  sheets  be  revisited  to 
ensure  that  they  properly  represented  each  BAFO.  With  this  requirement  in 
mind,  Area  Chiefs  exercised  strict  control  over  the  amount  of  effort 
expended  in  establishing  scores  above  the  lowest  levels  in  the  evaluation 
structure.  For  example,  evaluating  RAM/ILS  (See  Figure  .1)  3nd  scoring  it 
should  be  restricted  to  the  weighted  factors  (such  as  Quality  Engineering 
and  LOG/MANPRINT)  without  rolling  them  into  the  elements  (Reliability  and 
Maintainability  or  Integrated  Logistics  Support)  until  BAFO.  In  the  case 
of  this  Board,  the  SSAC  requested  a  preliminary  reading  on  scores  to  be 
presented  at  SSAC-II1.  Even  then,  complete  narratives  at  the  element 
level  or  area  level  were  not  necessary.  To  avoid  a  complete  rework  at 
BAFO,  it  is  recommended  that  only  preliminary  scores  (based  on  the  factor 
level)  and  strengths,  weaknesses,  green  flags,  red  flags,  and  risk  be 
documented.  While  all  of  these  findings  will  be  subject  to  change,  they 
will  focus  the  team's  effort  at  the  time  of  BAFO,  minimize  the  amount  of 
effort  required  to  document  the  findings,  and  still  provide  the  SSEB  with 
all  necessary  detail  to  interface  effectively  with  the  SSAC. 

In  executing  the  steps  in  the  process,  it  became  clear  that  scoring  below 
the  factor  level  should  be  avoided.  Instead,  it  is  recommended  that  the 
factors  include  topics  to  be  considered  in  arriving  at  a  factor  level 
score,  without  attempting  to  identify  subfactors  or  sub-subfactors,  which 
would,  otherwise,  force  a  significant  and  wasteful  increase  in  documenta¬ 
tion  time  and  effort.  At  the  same  time,  it  should  be  noted  that  this 
suggestion  will  not  affect  the  final  score,  so  long  as  the  necessary 
topics  are  included  at  the  factor  level. 

The  scoring  system  used  in  the  evaluation  was  based  on  awarding  a  passing 
grade  of  eight  on  a  scale  of  zero  to  ten.  Therefore,  an  offeror's  propos¬ 
al  stood  the  risk  of  being  penalized  significantly  if  requirements  were 
not  met.  At  the  same  time,  while  the  approach  supported  requirements  of 
the  RFP,  it  required  a  truly  exceptional  proposal  to  be  awarded  a  full  ten 
points  (two-pomt  bonus).  It  must  be  noted  that  in  the  case  of  this  SSEB, 
all  proposals  met  requirements  and  all  were  acceptable.  It  is  recommended 
that  the  spread  o.t  possible  points  above  a  passing  grade  be  increased 
slightly,  reflecting  a  change  to  a  score  of  seven  out  of  a  possible  ten. 
This  change  would  allow  the  details  of  the  evaluation  to  show  a  wider 
discrimination  among  good  proposals. 

The  initiative  taken  to  publish  weighting  of  areas,  elements,  and  factors 
in  the  RFP  (Figure  3)  initially  was  regarded  with  some  caution.  Without 
question,  it  provided  an  unmistakable  means  to  communicate  the  importance 
of  the  government's  new  initiatives.  It  is  highly  recommended  that  this 
methodology  be  adopted  in  all  future  procurements. 
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H.  MEETINGS  WITH  SOURCE  SELECTION  ADVISORY  COUNCIL 


This  SSEB  had  tour  separate  meetings  with  the  Source  Selection  Advisory 
Council  (SSAC’.  1  he  schedule  tor  these  meetings  is  shown  in  Figure  U. 

For  the  TSuO.  SSAC  members  were  assigned  responsibility  by  the  Chairman  to 
interface  with  the  SSEB  in  their  respective  specialty  areas.  Therefore, 
separate,  individual  meetings  with  SSAC  members  were  scheduled  to  ensure 
that  the  SSEB  took  maximum  advantage  of  exchanges  throughout  the  evalua¬ 
tion  process.  In  total,  about  JO  separate  meetings  were  held  between  SSAC 
and  SSEB  members.  Each  meeting  proved  to  be  an  excellent  forum  for 
promoting  dialogue  and  understanding  on  issues  and  led  to  a  smooth  flow  of 
information  preparatory  to  the  formal  SSAC  meetings  (SSAC-II,  -III,  -IV). 

The  four  major  SSAC  meetings  were  spaced  roughly  a  month  apart.  A  brief 
description  of  each  follows. 

1.  SSAC- I  -  27  MARCH  1985 

Objectives:  o  Remind  SSAC  of  RFF  requirements  in  each  area 

o  Describe  proposals  (area  bv  area)  received  from 
industry 

o  Identify  potential  issues 
o  Describe  plans  for  evaluating  proposals 
o  Describe  contracting  initiatives 

Comments-  This  meeting  was  the  SSAC's  first  exposure  to  the 
proposals.  It  included  brief,  20-minute  video  tapes  provided  by  each 
offeror  (serving  as  executive  summaries  of  their  proposals).  Also  avail¬ 
able  to  the  SSEB  and  SSAC  were  90-minute  video  tapes  (not  shown  at  SSAC-1) 
which  described  the  offerors'  proposals  in  greater  detail.  Overall,  the 
SSEB  presentations  and  the  offerors'  summary  tapes  required  about 
3*s  hours,  and  all  indications  were  that  the  information  passed  to  the  SSAC 
met  the  SSEB's  objectives.  Use  of  the  offerors'  20-minute  video  tapes  to 
augment  the  written  proposals  was  an  excellent  way  to  present  an  enormous 
amount  of  information  in  a  short  time.  It  was  an  SSEB  recommendation  to 
not  show  the  longer  (90-minute)  tapes  to  the  SSAC  because  they  were  not 
nearly  so  useful,  and  it  is  recommended  that  future  SSAC-SSEB  exchanges 
and  interfaces  concentrate  only  on  a  short  version  to  introduce  the 
proposa  Is  . 

2.  SSAC-II  -  23  APRIL  1985 

Objectives:  o  Identify  strengths  and  weaknesses  of  proposals 

(area  by  area) 

o  Describe  concerns  and  basis  of  errors,  omissions 
and  i lari f icat ions  required  from  each  offeror 


o  Report  assistance  provided  through  interfaces 
with  SSAC 

o  Describe  plans  to  be  executed  before  next  meeting 
of  SSAC 

Comments:  This  second  meeting  between  the  SSAC  and  SSEB  was 

highly  productive  and  formed  the  basis  for  clarifying  issues  with  the 
offerors.  By  the  time  of  this  meeting,  the  process  of  cleaning  up  errors 
and  omissions  was  well  underway,  and  fact-finding  discussions  with  the 
offerors  had  begun.  Because  of  the  1 nnova t ic ns  sought  through  this  RFP, 
unforeseen  difficulties  arose  in  evaluating  contractual  plans  provided  hv 
the  offerors.  It  will  be  recalled  that  the  RFP  did  not  dictate  details  of 
the  proposed  programs;  therefore,  the  basic  content  of  these  plans  was 
left  to  the  offeror’s  initiative.  The  approach  resulted  in  plans  that 
differed  in  substance  from  one  proposal  to  another,  causing  more  time  and 
effort  in  face-to-face  discussions  with  the  offerors  than  had  been 
planned.  With  these  new  concerns  in  mind,  the  SSEB  was  forced  to  conduct 
parallel  sessions  with  the  offerors  so  as  to  avoid  slippage  in  the  planned 
award  date.  A  change  in  schedule  details  (but  without  a  change  in  award 
date)  was  developed  for  this  SSAC  meeting,  and  the  revisions  were 
approved . 

3.  SSAC-III  -  22  MAY  1985 

Objectives:  o  Develop  results  of  preliminary  evaluation  and 

scoring 

o  Describe  progress  in  fact-finding  discussions  and 
negotiations 

o  Prepare  overview  of  potential  model  contracts 

o  Report  assistance  afforded  by  interfaces  with  SSAC 

o  Describe  plans  for  completion  of  evaluation  to  he 
presented  at  final  SSAC  meeting 

o  Describe  plan  for  down-select  to  one  contractor 
team  at  time  of  Preliminary  Flight  Rating  (PFR), 
and  seek  approval  from  SSAC  to  notify  offerors 


Comments:  The  third  SSAC  meeting  contained  considerable  detail 

and  provided  a  summary  of  potential  problems  that  needed  attention.  Based 
on  continued  individual  interfaces  between  the  SSAC  and  SSEB,  the  exchang¬ 
es  during  this  meeting  were  extremely  fruitful  and  produced  a  clear 
understanding  of  the  remaining  tasks.  It  was  recognized  by  all  that  the 
preliminary  scores  shown  were  subject  to  change  when  Best  and  Final  Offers 
were  evaluated  before  the  final  SSAC  meeting  (SSAC-1V).  Approval  was 
gLven  to  notify  the  offerors  of  criteria  to  fie  used  at  the  PFR 
down-select,  and  there  were  no  critical  issues  that  would  require  special 
attention.  The  remaining  schedule,  likewise,  was  approved,  and  the  SSEB 
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was  instructed  to  proceed  with  notifying  offerors  on  a  due  date  for  Best 
and  Final  Offer. 

4.  SSAC-IV  -  1  JULY  1985 

Objectives:  o  Present  final  evaluation  and  scores  (based  on 
Best  and  Final  Offer) 

o  Describe  strengths  and  weaknesses  of  proposals 
(area  by  area) 

o  Make  recommendations  for  approval  by  the  SSAC 

Comments:  This  final  meeting  with  the  SSAC,  again,  provided  a 
great  deal  of  detail  in  describing  the  basis  for  the  findings.  After  some 
discussion  and  review  of  lower  level  details,  recommendations  made  by  the 
SSEB  were  approved,  and  plans  were  set  into  motion  to  brief  the  SSA. 
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AUDIT  TRAIL 


As  with  any  similar  board  activity,  it  is  essential  that  all  steps  in  the 
selection  process  be  documented  to  support  findings  or  to  recount  events 
for  learning  purposes.  With  respect  to  this  SSEB,  an  audit  trail  is 
believed  to  have  added  significance,  particularly  because  of  the  new 
initiatives  and  innovations  identified  in  the  RFP.  Systems  employed  by 
the  Board  are  described  in  Section  II  of  this  after-action  report.  Other 
documents  generated  to  complete  the  audit  trail  are  discussed  in  the 
paragraphs  below.  In  addition,  it  is  the  intention  of  the  U.S.  Army 
Materiel  Command  to  disseminate  basic  content  of  the  contracts  along  with 
this  document  so  as  to  permit  applications  of  the  principles  to  other 
a  cquis i t ions . 


A.  SSEB  BOARD  REPORT 


The  Evaluation  Plan  details  the  procedures,  methods,  structure, 
documentation  required  of  the  SSEB,  The  Board  Report  is  the  cu1  ,  tion 
of  the  documentation  process  that  includes  the  subfactor,  factoi  .  lement, 
and  area  narratives  and  scoring.  For  the  T800 ,  this  report  was  developed 
in  iterative  steps  and  reflected  findings  from  the  initial  proposal,  the 
EOC  process,  fact  finding,  negotiations,  and  the  Best  and  Final  Offer 
(.BA FO) .  However,  as  discussed  previously,  BAFO  supersedes  all  earlier 
findings  and  is  the  basis  for  the  SSEB  recommendations  presented  to  the 
SSAC . 

The  SSEB  Board  Report  contains  four  Volumes.  Volume  I  is  intended  to  be 
an  executive  summary  and  a  comparative  analysis  of  the  proposals,  includ¬ 
ing  overall  Board  findings.  Volumes  II,  III,  and  IV  contain  the  board 
findings  down  to  the  subfactor  level;  one  volume  for  each  offeror. 

The  outline  of  the  Board  report  for  the  T800  SSEB  is  shown  below. 

BOARD  REPORT  OUTLINE 


VOLUME  I 


o  Executive  Summary 


o  Roster  of  SSEB 


o  Source  Selection  Evaluation  Plan 


o  Professional  Qualifications  of  Evaluators 


o  Comparative  Analysis 


VOLUMES  II,  III,  and  IV  -  The  remaining  volumes,  one  covering  each  of  the 
three  offerors  are  identically  structured  and  contain  the  specific  scores 
and  summaries. 


o  Introduction 

o  Evaluation  Sheets/Narratives/Scores  -  These  working  papers  precise¬ 
ly  cover  the  paragraphs  of  the  Evaluation  Plan  in  sequence,  beginning  at 
the  lowest  levels  scored  and  rolling  successively  through  Factors,  Ele¬ 
ments,  and  Areas. 

B.  SSEB  AFTER-ACTION  REPORT 

This  document,  which  presents  the  activities  of  the  T800  SSEB,  is  intended 
to  provide  guidance  to  others  who  may  become  involved  in  a  similar  source 
selection  process.  Each  board  will  seek  its  own,  most  effective  methodol¬ 
ogy;  therefore,  it  is  not  the  purpose  here  to  define  how  it  must  be  done. 
The  value  of  this  document,  instead,  is  in  lessons  learned  and  in  surfac¬ 
ing  potential  obstacles. 

Although  this  SSEB  attempted  to  use  lessons  and  recommendations  from 
previous  boards,  the  departure  from  past  practices  made  these  earlier 
reports  less  useful  than  had  been  hoped.  The  approach,  the  innovations, 
and  the  new  initiatives  required  by  the  RFP,  placed  many  all-new  concerns 
before  the  Board.  These  aspects  are  described  throughout  this  report  and 
will  not  be  repeated  here.  Many  of  the  topics  discussed  are  representa¬ 
tive  of  thoughts  expressed  jointly  by  the  offerors,  SSAC  members,  and 
participants  in  the  evaluation.  Specific  references  to  these  common  views 
are  found  in  Sections  II,  IV,  and  V. 

C .  DEBRIEFS 

1.  Losing  Offeror  -  The  debrief  was  conducted  for  the  unsuccessful 
offeror  in  accordance  with  FAR  15.1003  at  his  request.  The  purpose  of  the 
debriefing  was  is  to  identify  strengths  and  weaknesses  of  his  proposal,  as 
compared  to  the  requirements  of  the  RFP.  The  debrief  was  not  intended  to 
reveal  relative  merits,  technical  standings,  contents,  evaluation  scoring, 
or  similar  information  derived  from  competing  offerors'  proposals.  It 
should  be  noted  that  FAR  restrictions  define  the  type  of  information  that 
can  be  revealed,  and  it  became  clear  in  the  T800  debriefing  that  the 
unsuccessful  offeror  would  have  found  the  meeting  much  more  valuable  if 
the  specific  level  of  weakness  or  strength  in  the  proposal  could  have  been 
disclosed.  Recognizing  that  this  concern  is  not  new  to  any  board  of  this 
type,  it  is  recommended  that  the  losing  offerors'  evaluation  scores  be 
revealed  during  the  debrief,  but  without  making  comparisons  of  any  kind  to 
the  other  offerors.  In  this  way,  the  unsuccessful  offeror  can  enhance 
his  understanding  of  areas  that  needed  improvement.  The  information, 
obviously,  can  be  of  significant  value  in  future  competitions.  Without 
this  exposure,  it  will  be  difficult  for  any  offeror  to  identify  the 
magnitude  of  corrections  necessary  or  the  direction  of  new  emphasis. 

Another  consideration  tor  future  SSEBs  is  the  potential  of  releasing 
portions  of  the  Evaluation  Plan  to  the  offerors  with  the  RFP.  As 
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discussed  earlier,  it  also  may  be  beneficial  to  provide  copies  of  regula¬ 
tions  and  pamphlets  with  the  RFP.  The  insight  should  give  the  offerors  a 
better  understanding  of  the  selection  process. 


In  summary,  the  level  of  information  provided  to  an  unsuccessful  offeror 
in  a  debrief  should  be  of  sufficient  detail  to  permit  him  to  recognize  his 
strengths  and  weaknesses  more  readily.  He  then  can  place  proper  emphasis 
and  effort  in  appropriate  areas  so  as  to  strengthen  future  proposals, 
which,  ultimately,  would  benefit  the  government. 


2.  Winning  Offeror(s)  -  A  debrief  normally  is  held  with  the  winning 
contractor  also,  and,  while  not  mandatory,  it  can  prove  to  be  highly 
beneficial  to  the  government,  as  well  as  to  the  contractor.  Its  purpose 
is  to  identify  areas  in  which  the  government  believes  that  the  contractor 
can  focus  additional  attention  during  performance  of  the  contract.  It  can 
be  an  excellent  forum  for  exchange  of  information  and  ideas,  although  it 
is  not  directive  in  nature.  For  the  T800,  the  expected  benefit  of  such  a 
briefing  is  that  the  contractor  very  likely  will  take  appropriate  actions 
to  be  in  a  more  favorable  competitive  position  at  down-select,  with  the 
government  being  the  ultimate  beneficiary. 


SECTION  IV 


HEW  INITIATIVES 


The  purpose  of  this  section  is  to  highlight  the  major  lessons  learned  and 
identify  key  positive  aspects  of  the  new  way  of  doing  business.  As  noted 
in  previous  sections  of  this  report,  the  structure  of  the  T800  SSEB  was  a 
major  departure  from  past  practices,  particularly  as  related  to  the 
evaluated  areas  and  their  respective  weighting.  Historically,  other  SSEBs 
were  structured  into  three  Areas  -  Technical  (weighted  70%),  Management 
(weighted  30%)  and  Cost  (evaluated  but  not  weighted  and  scored).  For  the 
T800  SSEB,  there  were  five  Areas  (Figure  3).  The  two  new  Areas  were 
RAM/ILS  and  Production  Competition,  both  of  which  were  a  direct  reflection 
of  the  Army's  dedication  to  key  LHX  Program  goals.  These  areas  also 
emphasized  the  importance  placed  on  affordability.  For  a  complete  de¬ 
scription  of  the  initiatives  fostered  by  this  break  with  tradition,  the 
reader  is  referred  to  the  after-action  report  on  the  T800  RFP,  An 
Innovative  Approach  to  a  Performance-Oriented  Request  for  Proposal,  dated 
10  December  1984.  That  report  described  the  rationale  and  approach  to 
evaluation  of  Production  Competition  and  RAM/ILS,  both  of  which,  for  the 
first  time,  where  considered  as  key  drivers  in  the  selection  process.  It 
was  regarded  with  the  utmost  importance  that  the  offerors  address  these 
new  areas  thoroughly  in  their  proposals,  much  more  so  than  in  any  previous 
acquisition.  More  specifically,  the  offeror  stood  the  risk  of  having  his 
proposal  rejected  if  Production  Competition  initiatives  were  not  addressed 
completely,  regardless  of  the  merits  associated  with  other  areas  of  his 
proposal.  The  impact  of  adding  Production  Competition  and  RAM/ILS  to  the 
evaluation  was  offset  somewhat  by  the  change  in  emphasis  on  the  Technical 
and  Management  Areas.  On  previous  SSEBs,  RAM  and  ILS  were  included  as 
elements  or  factors  under  the  Technical  and  Management  Areas.  However, 
both  new  Areas  contained  subject  matter  that  had  not  been  evaluated  by  any 
previous  SSEB.  In  addition.  Cost  was  being  scored  on  the  basis  of  firm 
fixed  price  guarantees  and  liabilities,  most  of  which  involved  far-reach¬ 
ing  commitments  never  before  required  at  the  outset.  Therefore,  the 
evaluation  truly  required  learning  on  the  part  of  both  the  evaluation  team 
and  the  offerors,  alike. 

Along  with  this  change  in  emphasis,  a  number  of  experienced  SSEB  special¬ 
ists  were  in  short  supply.  Many  board  members  (approximately  90  percent 
of  the  RAM/ILS  evaluators)  had  never  served  on  an  SSEB.  Coupled  with  the 
many  new  initiatives  in  this  Area,  occasional  lengthy  and  detailed  train¬ 
ing  sessions  were  required,  not  just  on  the  process,  evaluator 

techniques,  and  administrative  procedures,  but  on  the  overall  conduct  of 
the. Board,  as  well. 

It  was  obvious  during  the  initial  stages  of  the  evaluation  that  several 
evaluators,  although  technically  competent  and  capable,  encountered 
difficulty  in  expressing  their  findings  in  clear,  concise  evaluation 
language.  A  prescribed  format  and  examples  of  filled  in  scoring  sheets 
with  narratives  would  have  minimized  early  inconsistencies  and  false 
starts.  To  preclude  this  concern,  a  training  session  for  all  members 
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should  he  conducte-,  it  least  Luo  days  heh'rc  proposals  are  received, 
regardless  .>t  their  past  participati  ui  n  SSKBs  .  Covered  in  this  training 
should  he  s  L  u:ida  id  i  zed  t  -rnt.its  i  examples  ot  proper  urite-ups  ami  storing 
procedures  ,  .  r  :  t  rr  :  .1  to  sir  nig  ami  weak  points  or  red  and  green  tlags, 
risk  assessment,  t  tie  V  process  and  procedures  tor  conducting 

ne  go  t 1  a  t  1 ons 


Eariv-o:i,  1  ma  ;  hurdle  was  L  >  understand  the  philosophy  ot  the 
pert  rnuiio'-'  rientej  n.piisition  method.  Idle  1  1  rst  tew  days  ot  the  hoard 
uere  used  t  reii  the  kfl’  ami  Source  Selection  Evaluation  Plan.  During 
this  time,  tm.se  Sud  members  who  had  been  involved  with  the  development 
t  the  kFR  worked  extensively  uith  the  other  members,  answering  questions 
md  discussing  t  tie  intent  ot  the  acquisition  method.  Downstream,  an 
overwhelming  maturity  ?  the  participants  embraced  the  new  method  and 
enl hus last ic  1  !  ! y  supported  it.  Clearly,  the  evaluators  needed  to  under¬ 
stand  that  the  tterors  were  given  the  opportunity  to  express  their 
approach  by  laying  .ut  detailed  5. Ians,  rather  than  parroting  hack  plans 
dictated  by  the  government  m  a  more  c lassical  KFP . 


this  t ime ,  t m >se 
't  the  kf  k  wo rKi* 


Because  it  was  the  otteror's  responsibility  to  evolve  and  then  commit  his 
plans,  details  developed  through  discussions  and  negotiations  required  a 
number  of  iterations.  In  addition,  several  issues  remained  open  from  one 
negotiation  session  to  the  next,  and  contracts  received  from  all  offerors 
at  BAFO  still  contained  varying  degrees  of  weaknesses.  For  the  T800 
program,  these  weaknesses  should  he  corrected  easily  during  the  competi¬ 
tive  phase  ot  development  prior  to  down-select.  However,  there  is  a 
legitimate  concern  that  if  only  one  contract  had  been  awarded,  there  might 
at  be  the  same  opportunity  to  review  and  renegotiate  issues.  The  govern- 
"it’M L  shell]  i  oevi.'i.jt  ■  the  process  to  assess  whether  a  follow-up  negotia- 
t;  n  s->ss  1  it  u  •  *.  i:  the  selected  wmner(s)  can  produce  the  desired  result 
*  .  L  'unit  a  negat  ive  impact  on  other  commitments. 

A.  MANAGEMENT  AREA 

The  Management  Area  was  reduced  in  scope  as  a  part  of  the  selection 
process  tor  a  number  of  reasons.  Having  separated  RAM  and  I LS  from  the 
Management  and  Technical  Areas,  it  became  clear  that  the  only  question  to 
he  addressed  was  whether  management  systems  in  place  in  industry  were  well 
enough  structured  to  exert  lse  necessary  controls  and  to  conduct  the  engine 
development  program.  Obviously,  all  offerors  brought  great  strength  and 
pi  uess  to  the  government  in  their  proposals.  Therefore  the  question  was 
little  significance  pa r 1 1 cu 1  a r 1 y  in  view  of  the  requirement  for  a  firm 
fixed  price  development  (ontract,  with  the  offerors  prepared  to  make 
long-range  commitments  and  guarantees.  In  addition,  most  issues  were  more 
closely  tied  to  the  other  ar».is  of  the  evaluation  structure. 

The  Management  Area  (Figure  bj  included  five  individuals  -  one  chief  and 
four  evaluators,  who  concentrated  on  how  teaming  was  implemented,  while 
maintaining  the  management  effectiveness  of  the  parent  organizations. 
Distribution  and  allocation  of  authority,  responsibility,  workload  and 
resources  were  examined,  along  with  communications,  data  flow,  and  checks 
and  balances  within  the  proposed  team  structure.  Two  of  the  Elements  - 
System  Engineering  Management  and  Configuration  Management  -  served  only 
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to  confirm  that  all  offerors  would  conduct  these  functions  in  the  tradi¬ 
tional  fashion. 


In  retrospect,  the  Management  Area  tended  to  be  more  like  business  as 
usual  than  it  should  have  been,  and  the  RFP  and  the  Evaluation  Plan  did 
not  go  far  enough  to  change  that  situation.  In  future  similar  programs, 
it  is  recommended  that  Management  be  eliminated  as  an  Area  and  that  the 
functions  be  conducted  in  each  of  the  other  areas  where  the  issues  are 
better  defined  and  more  specific.  If,  for  purposes  of  an  overview, 
Management  should  be  retained  as  an  Area,  it  should  not  be  weighted  and 
scored  and  should  be  used  only  to  ensure  that  there  are  not  major  manage¬ 
ment  shortfalls. 


B.  PRODUCTION  COMPETITION  AREA 


The  T800  SSEB  was  the  first  board  to  evaluate  Production  Competition  as  a 
separate  Area.  Never  before  had  the  Army  attempted  to  obtain  such  long¬ 
term  commitments  to  competition  this  early  in  a  program.  Clearly,  estab¬ 
lishment  of  this  Area  was  a  precedent-setting  break  from  business  as 
usual,  and  the  resulting  visibility  served  to  demonstrate  the  Army's 
dedication  to  developing  affordable  weapon  systems. 


The  Production  Competition  Area  (Figure  7)  was  divided  into  three 
Elements  -  End  Item,  Parts,  Producibility  Engineering  and  Planning  (PEP). 
The  area  was  staffed  with  19  people  from  a  variety  of  backgrounds.  Like 
the  RAM/ILS  Area,  there  was  some  early  concern  for  locating  and  assigning 
experienced  people  to  staff  the  effort.  Because  this  Area  of  the  evalua¬ 
tion  was  completely  new,  no  comparisons  can  be  made  with  past  practices. 
However,  the  innovative  aspects  of  the  T800  contracts  related  to  Produc¬ 
tion  Competition  Area  are  extensive.  In  view  of  the  foregoing  discussion 
on  the  Management  Area,  it  is  suggested  that  a  better  structure  for 
Production  Competition  would  have  been  as  shown  in  Figure  8. 


Among  the  more  significant  departures  from  past  practices  derived  from 
the  T800  RFP  were  readiness  and  production  competition  initiatives.  In 
the  latter  area,  offerors  were  expected  to  address  the  following  subjects 
in  their  proposals. 


o  Competition  in  Research  and  Development 


o  Alternate  Production  Sources 


o  Facilities  Provided 
o  Competition  Among  Subcontractors 
o  Expanded  Base  of  Vendors  and  Suppliers 
o  Technical  Data  Package  (TDP)  Options 
o  Additional  Vendor  Qualification  Options 
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Proactive  Socioeconomic  Initiatives 
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.1  New  or  Better  Ideas  to  Enhance  Competition 

It  was  intended  that  as  a  result  of  this  early  emphasis  on  production 
competition,  winning  contractors  would  strive  to  maximize  their  dedication 
to  executing  plans,  improving  production  ana  cost  commitments,  and  meeting 
or  surpassing  schedule  requ 1 rement s  while  conducting  technical  develop¬ 
ment.  In  the  case  of  the  T800,  the  competing  teams  are  aware  that  based 
on  overall  performance  and  progress,  a  government  down-select  to  one 
contractor  team  after  about  3  years  of  FSD  lies  ahead.  Therefore,  a  team 
must  demonstrate  that  it  is  best  prepared  to  continue  development  through 
qua  1  i  f icat ion  and  be  in  a  ready  state  to  enter  production.  A  critical 
point  in  the  schedule  beyond  FSD  is  at  Lot  III  in  production  (2  years 
after  qua  1 1 f l cat  ion  )  ,  where  the  companies  which  make  up  the  down-selected 
team  will  compete  to  deliver  the  same  engine.  The  T800  Acquisition 
Schedule/Mi lestones  are  shown  in  Figure  2. 


A  unique  feature  if  the  T800  engine  RFP  lies  in  the  requirement  for 
technical  transfusion  during  FSD  and  the  establishment  of  a  second  produc¬ 
tion  source  of  the  same  design  for  the  successful  team.  It  is  important 
to  realize  that  contractors  have  agreed  to  exchange  the  necessary  technol¬ 
ogy  and  know-how  in  a  contractual  agreement  established  for  the  end  item 
portion  of  the  Production  Competition  Plans.  These  innovative  plans  are 
contractually-binding  attachments.  Agreements  have  been  negotiated  to 
define  practices  and  procedures  to  achieve  the  technical  interchange 
necessary  to  enable  production  of  the  selected  single  engine  design  at 
each  of  the  two  facilities  of  the  down-selected  team.  At  the  end  of  the 
development  program,  both  must  manufacture  and  test  that  engine  (300-hour 
test)  to  demonstrate  the  ability  of  each  to  deliver  independently. 


This  key  competition  facet  of  the  program  is  also  a  strong  readiness 
feature.  The  Army  now  has  prepared  itself  to  enter  production  with  two 
engine  sources,  and  with  complete  interchangeability  of  all  parts  and 
components.  Regardless  of  the  winner  at  down-select,  the  developer  team  is 
contractually  committed  to  maintain  interchangeability  and  have  agreed  to 
establish  an  Interfacing  Material  Review  Board  for  nonconforming  material, 
Class  1  design  changes.  Class  II  design  changes  and  Tooling  Data  Bases. 
Periodic  int e rchangeah i 1 i ty  demonstrations  will  be  and  are  an  important 
factor  in  ensuring  maintenance  of  the  common  design.  The  required  common¬ 
ality  will  he  maintained  through  agreements  covering  procedures  to  control 
Engineering  Change  Proposals  (ECPs)  for  Class  I  changes.  With  two  produc¬ 
tion  sources,  it  also  appears  certain  that  there  will  be  fewer  Class  1 
changes  than  in  the  past.  This  observation  is  based  on  the  belief  that 
some  proposed  changes,  which  may  be  associated  with  advantages  to  a  single 
contractor  (because  of  a  unique  technical  or  manufacturing  situation)  will 
be  avoided  altogether.  This  self-imposed  control  and  configuration 
stability  sought  by  two  manufacturers  also  is  expected  to  result  in 
earlier  maturation  of  the  configuration,  which  will  yield  improvements  in 
productivity  and  unit  cost.  It  follows  that  the  contracts  are  constructed 
to  establish  anil  maintain  production  competition  between  two  prime  sources 
throughout  the  life  cycle  of  the  T800  engine  program.  The  concept  will 
provide  for  a  lower  tier  production  base  to  support  surge  and  mobilization 


spare  and  repair  parts,  as  well  as  two  sources  of  supply  for  end  item 
engine  production. 

'■...uet  Key  element  written  into  the  contracts  is  that  the  government  will 
not  directly  fund  facilitization  of  the  T800  engine,  nor  such 
previously-funded  items  as  special  tooling  and  test  equipment.  The 
approach  is  to  require  industry  to  make  these  coiranitments ,  including  any 
related  requirements  in  the  vendor  and  supplier  chain.  A  fallout  of  this 
approach  is  that  planning  for  test  equipment,  tooling  and  facilitization 
will  be  integrated  carefully  with  overall  corporate  investments  and  will 
force  industry  to  apply  controls  over  the  development  community,  including 
consideration  of  nondevelopmenta 1  items  where  little  or  no  facilitation 
will  be  required. 

With  company  financial  commitments  at  all  levels  in  the  subcontractor/ 
vendor  chain,  it  is  expected  that  initial  vendors  and  suppliers  will 
remain  active  as  sources  of  supply  throughout  the  life  cycle,  offering 
cost  and  readiness  advantages.  The  stability  in  the  vendor  base  will 
offer  a  significant  capability  for  the  Army  to  surge  and/or  acquire 
mobilization  quantities. 

The  innovations  fostered  will  yield  a  significant  improvement  in  parts 
competition  (two  sources  minimum  for  all  parts)  when  the  engine  enters 
production.  The  present  unfavorable  situation  in  mature  weapon  systems, 
as  well  as  in  initial  production  items,  requires  that  the  government  and 
contractors  convert  sole-source  parts  to  competition  by  breakout,  part  by 
part,  at  considerable  cost  to  both.  This  situation  will  be  avoided  for 
all  practical  purposes  with  T800  contracts.  Should  any  form  of  breakout 
and  parts  management  (because  of  high  cost  parts)  be  required,  the  con¬ 
tracts  already  contain  priced  options  to  buy  Technical  Data  Packages  and 
qualify  alternate  vendors.  In  effect,  a  fallback  position  already  is 
written  into  the  contracts. 

To  achieve  intended  objectives  at  the  beginning  of  production,  the  con¬ 
tract  requires  competition  beyond  the  prime  contractors  own  bill  of 
material.  Because  the  baseline  engine  is  considered  to  be  mature  at  that 
point,  the  contractor  can  elect  to  propose  parts  for  an  option  to  acquire 
Technical  Data  Packages  (TDPs).  Firm  TDP  and  AVQ  option  prices  are 
established  in  the  contracts  which  can  be  exercised  through  delivery  of 
the  last  hardware  in  the  fifth  production  (Lot  V)  contract.  Also  the 
contracts  specifically  identify  the  type  of  rights  associated  with  each 
TDP.  However,  the  rights  specified  must  meet  the  Government's  minimum 
needs  for  competitive  reprocurement  of  parts  in  order  to  have  enhanced 
the  contractor’s  proposal  and  gained  a  positive  evaluation.  In  addition 
to  any  proposed  TDPs,  the  contracts  include  a  provision  which  takes  effect 
a  fixed  period  after  the  award  date  of  the  FSD  contract,  and  grants  the 
Government  additional  rights  in  any  limited  rights  technical  data 
deliverable  under  this  or  any  future  related  contract.  These  additional 
rights  permit  the  Government  to  disclose  the  data  to  a  third  party  to 
manufacture  replenishment  parts,  subject  to  a  restriction  against  further 
disclosure  by  the  recipient.  The  Governments  responsibility  to  the  contractor 
is  limited  to  providing  the  data  to  the  recipient  with  the  appropriate 


restrictions.  This  provision  has  particular  importance  for  replenishment 
parts  which  do  not  require  detailed  process  and  manufacturing  specifications 
to  manufacture  the  part,  such  specifications  not  normally  being  deliverable 
items  under  the  contract. 

For  parts  not  included  in  the  contractor's  own  bill  of  material  and  for 
parts  not  identified  for  the  TDP  option,  the  contractor  has  proposed  to 
qualify  alternate  vendors  at  government  expense.  However,  a  firm  fixed 
price  or  not  to  exceed  price  for  the  qualification  of  these  additional 
sources  already  has  been  negotiated  and  established  in  the  contract. 

Closely  connected  to  the  keystone  of  competition  in  the  contractor's  bill 
of  material  is  the  requirement  that  second  sources  be  qualified  to  the 
same  specifications  and  in  the  same  manner  as  the  original  source. 

Concern  related  to  imposing  these  requirements  on  small,  small  disadvan¬ 
taged  and  women  owned  businesses  caused  the  Army  to  seek  an  assurance  of 
fairness  in  the  qualification  of  additional  sources.  Therefore,  in  the 
development  and  execution  of  his  plans,  the  contractor  is  required  to  have 
available  as  the  second  source,  small,  small  disadvantaged  and  women  owned 
businesses  or  prove  that  such  sources  of  supply  do  not  exist  or  are  not  in 
the  best  interest  of  the  Army.  This  approach  is  proactive  and  requires 
the  contractor  to  canvass  the  domestic  industrial  base  to  locate  and 
solicit  such  suppliers.  For  the  contractors,  subcontractors,  vendors  and 
suppliers,  it  is  extended  to  include  the  alternate  vendor  qualification 
option . 

Yet  another  innovation  within  the  contracts  is  the  challenge  to  industry 
to  be  responsive  by  offering  unique  business  proposals  intended  to  achieve 
the  goals  and  objectives  that,  historically,  were  accomplished  by  breakout 
and  alternate  vendor  qualification.  Proposals  contained  in  the  contracts 
consider  alternatives  to  the  government’s  acquisition  of  technical  data, 
management  of  the  configuration,  procurement  of  breakout  hardware,  and 
administration  of  alternative  source  contractors  (as  well  as  government 
cost  attended  to  those  efforts).  New  ideas  have  been  offered,  and  the 
Army  now  must  address  changes  to  historic  ways  in  which  business  has  been 
conducted.  During  this  first  36  months  of  FSD,  the  Army  must  fully 
develop  the  potential  for  implementing  (partially  or  completely)  the 
contractually-offered  new  ideas . 

The  T800-XX-800  RFP,  the  Evaluation  Plan,  and  the  subsequent  SSEB  actions 
have  enabled  the  Army  to  obtain,  for  the  first  time,  contracts  with 
industry  which  contain  a  binding  plan  for  establishment  and  maintenance  of 
vendor  and  supplier  competition  for  engine  parts  in  an  expanded  industrial 
support  base.  The  Competition  Plan  was  negotiated  in  great  detail  to 
provide  firm  commitments  and  milestones,  as  well  as  challenging  goals. 

C.  TECHNICAL  AREA 

The  Technical  Area  of  the  SSEB  was  structured  as  shown  in  Figure  9. 
Essentially,  it  was  the  same  as  for  previous  engine  SSEBs .  However,  it  is 
important  to  not*-  that  the  number  of  evaluators  was  reduced  by  about  25 
percent,  largely  because  of  extensive  use  of  computer  systems  to  perform 
such  analyses  as  hearing  life,  engine  performance  predictions,  and 
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structural  integrity  (as  opposed  to  manual  systems  or  desk-top  calcula¬ 
tors).  In  reviewing  the  structure  of  this  Area,  it  is  suggested  that  in 
future  RFP/SSEB  activities,  Survivability  and  Vulnerability  be  moved  from 
RAM/ILS  to  Technical  as  this  subject  is  more  closely  aligned  to  topics 
covered  under  design  and  development.  This  change,  along  with  the  previ¬ 
ously  discussed  shift  away  from  the  Management  Area,  is  shown  in 
Figure  10.  Other  lessons  learned,  recommended  improvements  and  consider¬ 
ations  for  future  procurement  actions  are  provided  below. 

In  the  T800  engine  acquisition  strategy,  the  performance-oriented  RFP 
represented  the  government's  intention  to  provide  industry  with 
flexibility  in  meeting  technical  requirements.  It  translated  into  dis¬ 
carding  the  "how-to-do-it"  dictates  of  former  RFPs  and  allowed  industry  to 
propose  a  best  plan  and  approach,  including  their  optimizing  of  trade¬ 
offs.  Having  now  evaluated  the  proposal  resulting  from  this  RFP,  several 
areas  of  caution  can  be  identified.  The  first,  and  probably  most  impor¬ 
tant,  is  that  although  there  is  wide  support  for  not  telling  the  offeror 
how  to  meet  design  requirements  or  put  together  his  program,  the  govern¬ 
ment  should  clearly  identify  the  kind  of  testing  and  substantiation  needed 
to  support  the  design.  Also,  care  must  be  taken  to  ensure  that  any  trade¬ 
offs  proposed  by  an  offeror  and  approved  by  the  evaluation  team  be  dissem¬ 
inated  to  all  offerors  so  that  a  fair,  competitive  environment  is 
maintained.  These  details  will  become  part  of  the  contractual  plans; 
therefore,  it  will  be  necessary  for  the  evaluators  to  use  them  in  develop¬ 
ing  risk  assessments.  It  has  been  the  perception  that  too  many  such 
details  were  contained  in  military  specifications.  On  the  other  hand,  it 
is  not  enough  to  tell  offerors  that  their  engines  must  ingest  birds  or 
operate  in  a  dusty  environment  without  defining  the  environments.  In  this 
example,  the  size,  weight,  and  speeds  at  which  a  bird  might  be  ingested, 
and  the  sand  concentration  and  particle  sizes  expected,  must  be  provided 
for  the  designer.  When  such  tjpics  are  not  addressed  in  the  RFP,  a 
question  certainly  will  arise  as  to  what  the  real  requirements  are.  The 
concern  also  carries  over  into  the  Evaluation  Plan  and  must  be  dealt  with 
clearly  to  provide  primary  evaluation  criteria.  There  must  be  no  ambigui¬ 
ty  between  the  RFP  and  any  follow-on  activity  (the  offeror's  proposal,  the 
government's  evaluation,  and  performance  of  contract). 

Regardless  of  the  number  of  military  specifications  eliminated,  a  large 
number  of  them  were  developed  on  the  basis  of  many  years  of  lessons 
learned.  Without  question,  many  should  be  discarded  as  not  necessary  or 
germane,  and  each  one  eliminated  will  require  some  explanation  or  re¬ 
training  for  evaluators  so  as  to  avoid  costly  and  time-consuming  effort. 
The  answer  may  be  to  develop  generic  specifications  that  can  be  tailored 
to  the  special  requirements  of  the  procurement  at  hand,  and  eliminate  all 
others.  It  is  recognized  that  it  will  be  the  offeror's  responsibility  to 
define  how  he  intends  to  meet  those  requirements.  We  should,  however, 
continue  to  define,  in  sufficient  detail,  the  pass-fail  criteria  for 
testing  necessary  to  substantiate  that  requirements  have  been  met. 
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0.  RAM/ILS  AREA 


The  establishment  of  RAM/ILS  as  a  separate  evaluation  area  was  a  bold  and 
innovative  break  with  tradition.  Previously,  the  elements  and  factors 
of  this  area  were  placed  with  the  Technical  and  Management  Areas  of  SSEBs . 
The  change  was  made  to  give  a  very  clear  signal  to  industry  that  the 
government  intended  to  view  RAM/ILS  as  equal  in  importance  to  any  other 
area  evaluated.  Therefore,  it  truly  became  a  potent  entity,  possessing 
the  same  influence  on  the  selection  as  Cost  and  Technical  Areas.  This  new 
status,  a  clear  break  from  business  as  usual,  was  reflected  in  the  RFP 
with  a  weighting  factor  of  30  percent. 

The  RAM/ILS  Area  was  structured  with  separate  RAM  and  ILS  Elements,  as 
shown  in  Figure  11.  They  were  weighted  60  and  40  percent,  respectively. 

The  RAM  element  was  divided  into  six  factors  (Reliability, 

Maintainability,  Hazard  Control,  Human  Factors,  Quality  Engineering,  and 
Survivability/Vulnerability)  while  the  ILS  Element  was  divided  into  two 
Factors  (Logistics/MANPRINT  and  Air  Vehicle  Support  (AVS)).  Each  element 
and  factor  was  headed  by  a  chief  whose  duty  was  to  provide  overall  direction 
and  management  of  his  particular  topic  of  evaluation.  At  the  height  of  the 
Board's  activity,  there  were  20  evaluators  in  the  RAM  Element  and  17  in  ILS. 

To  enhance  the  interaction  among  evaluators  of  Maintainability  and  Human 
Factors  in  the  RAM  Element  and  interface  the  maintenance  concepts  with 
MANPRINT  evaluators  in  the  ILS  Element,  a  structure  that  would  have  led  to 
a  better,  more  effective  integration  of  RAM  and  ILS  is  shown  in  Figure  12. 

It  depicts  a  structure  that  also  provides  higher  visibility  to  MANPRINT. 

It  will  be  noted,  as  discussed  earlier,  Survivability/Vulnerability  have 
been  shifted  to  the  Technical  Area.  Other  changes  from  Figure  11  are  in 
the  weighting  of  factors  under  RAM.  Because  of  the  importance  that 
ultimately  will  be  placed  on  the  Quality  Engineering  Factor,  and  consider¬ 
ing  the  requirements  and  plans  that  must  be  evaluated,  it  is  recommended 
that  it  be  afforded  higher  weighting  in  future  SSEBs. 

The  structure  used  for  the  performance-oriented  RFP  surfaced  a  significant 
problem  during  evaluation  of  the  required  RAM/ILS  plans.  Because  require¬ 
ments  were  stated  in  several  different  sections  of  the  RFP  and  the  Systems 
Specification,  the  offerors  provided  information  vital  to  the  RAM/ILS 
evaluation  throughout  their  proposals,  and  much  of  this  information  was 
not  cross-referenced  to  the  RAM/ILS  volumes.  Apparently,  offerors  assumed 
that  the  ILS  evaluators  had  read  all  parts  of  all  proposals.  Therefore, 
in  the  initial  stages  of  the  evaluation,  vital  topics  might  have  been 
missed  by  evaluators  who  were  not  aware  of  the  content  of  other  volumes. 

Some  unnecessary  EOCs  and  lengthy  discussions  with  the  offerors  that 
followed  could  have  been  avoided  if  a  cross-reference  system  had  been 
required  by  the  RFP.  For  the  benefit  of  all,  future  RFPs  should  provide 
clear  instruction  that  the  offeror  should  repeat  or  otherwise  identify  all 
material  believed  to  be  germane  in  every  volume  of  his  proposal.  It 
should  be  the  offeror's  responsibility  to  ensure  that  as  a  minimum,  every 
volume  is  cross-referenced  so  that  the  evaluators  can  track  the  informa¬ 
tion  in  the  manner  that  the  offeror  wishes. 


E .  COST  AREA 

Early  in  the  formulation  of  the  RFP,  it  was  determined  that  the  Cost  Area 
would  be  scored.  This  was  a  major  departure  from  previous  source  selec¬ 
tions,  where  cost  was  estimated  and  presented  by  evaluators,  but  not 
scored.  The  Cost  Area  was  weighted  30%  of  the  total  T800  evaluation,  and, 
as  shown  in  Figure  13,  there  were  three  elements  defined  and  weighted  - 
Research,  Development,  Test,  and  Evaluation  RDT&E  (20%),  Procurement 
(40%),  and  Operating  and  Support  (40%).  Following  the  overall  tone  of  the 
RFP,  the  Cost  Area  instructions  were  brief  and  performance-oriented.  The 
page  count  of  the  Cost  section  of  the  T800  RFP  was  two  pages,  compared  to 
approximately  30  pages  in  other  solicitations.  In  this  evaluation,  the 
Procurement  and  Operating  and  Support  (O&S)  Cost  Elements  did  not  concen¬ 
trate  on  comparisons  against  a  complete  and  documented  government  estimate 
of  the  offeror's  data.  Instead,  government  goals  were  set  for  Procure¬ 
ment,  also  known  as  Design-to-Cost  (DTC),  and  0&S  Cost.  In  this  break 
from  business  as  usual,  offerors  were  required  to  propose  contractually- 
binding  guarantees  in  support  of  their  own  definitive  commitments.  In  the 
spirit  of  this  performance-oriented  acquisition,  all  details  on  how  to 
propose  the  DTC  and  O&S  guarantees  were  left  entirely  to  the  offeror's 
option.  Despite  this  obvious  brevity  in  the  RFP,  offerors'  responses  were 
very  detailed.  Even  greater  detail  was  developed  with  the  offerors  in 
follow-on  discussions  during  the  evaluation  period.  RDT&E  Cost,  as 
required  by  the  RFP,  was  offered  as  a  Firm  Fixed  Price  (FFP) ,  with  the 
exception  of  an  Air  Vehicle  Support  line  item,  which  was  requested  on  a 
cost-type  basis.  For  this  RDT&E  Element,  a  considerable  amount  of  depth 
and  back-up  detail  was  required  to  assess  labor,  materiel  and  burden  in 
the  Work  Breakdown  Structure  (WBS) .  Therefore,  the  RFP  required  that  such 
detail  be  provided  on  computer  floppy  disks,  using  LOTUS,  1,  2,  3, 
software  -  another  innovation  that  proved  to  be  most  helpful  lo  the 
evaluators  and  will  be  equally  beneficial  to  the  LHX  Project  Manager  after 
contract  award. 


Overall,  the  Cost  Area  was  assigned  eight  full-time  and  six  part-time 
analysts,  which  was  about  half  the  usual  number  for  a  major  source  selec¬ 
tion.  One  reason  for  using  less  manpower  was  that  the  RDT&E  cost  was 
offered  on  a  competiti,?,  FFP  basis,  and,  based  on  predetermined  indepen¬ 
dent  government  estimates,  all  offerors  were  well  within  goals.  There 
simply  was  less  need  for  arduous  changes  after  detailed  cost  negotiations. 
Secondly,  use  of  desk-top  personal  computers  and  accompanying  software 
relieved  the  Board  of  performing  detailed  spread-sheet  calculations.  In 
the  future,  when  data  are  required  on  tape/disk,  consideration  should  be 
given  to  requesting  one  hard  copy  of  the  data,  as  well. 

It  was  envisioned  that  the  evaluation  of  guarantees  for  bi'C  and  O&S  cost 
would  be  less  time  consuming  than  the  traditional  requirement  to  perform  a 
complete  independent  government  estimate  for  these  elements.  In  reality, 
the  reverse  was  true,  and,  indeed,  it  was  necessary  to  conduct  a 
detailed  analysis  for  each  offeror  and  to  interface  with  the  legal  advisor, 
Technical  and  RAM/ILS  personnel.  Perhaps,  a  generic  cost  model  could  have 
been  developed  early-on  to  cover  an  estimated  range  of  DTC  and  0&S  costs. 
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but  the  more  correct  approach  proved  to  be  that  maximum  flexibility  would 
be  achieved  by  permitting  specific  features  of  each  engine  to  influence 
the  offeror's  proposal.  To  maintain  this  flexibility,  however,  detailed 
attention  was  given  to  these  elements  throughout  discussions  and  negotiations, 
and  considerable  analysis  was  necessary.  Therefore,  different  from  RDT&E , 
more  manpower  than  expected  was  required  to  evaluate  Procurement  and  O&S 
Costs.  In  future  SSEB  activities,  where  the  same  approach  may  be  used,  it 
is  recommended  that  at  least  two  economists  be  available  (one  full-time, 
one  part-time)  to  project  the  impact  of  guarantees  on  budgets  in  the  out 
years.  The  need  is  for  a  very  precise  and  explicit  escalation  analysis. 

In  short,  the  new  and  innovative  approach,  which  shifted  cost  assessments 
toward  evaluation  of  guarantees,  probably  did  not  yield  a  net  manpower 
reduction  for  the  Procurement  and  O&S  Elements.  The  time  required  to 
perform  the  evaluation,  as  compared  to  past  SSEBs ,  was  judged  to  be  about 
equa 1 . 

As  discussed  earlier  in  this  report,  the  last  days  for  the  evaluation 
(between  BAFO  and  the  final  SSAC  briefing)  were  extremely  busy  and  over¬ 
loaded  the  entire  SSEB.  It  is  strongly  recommended  that  at  least  2  weeks 
be  provided  to  future  SSEBs  to  avoid  a  hectic  situation  involving  prepara¬ 
tion  of  several  detailed  briefings  and  completion  of  the  final  analysis  at 
the  same  time.  This  was  particularly  true  in  the  Cost  Area,  where  a 
complete  reverification  of  the  offerors’  data  was  essential  to  produce 
accurate  and  reliable  final  scores.  Obviously,  the  time  required  will  be 
highly  dependent  on  the  number  of  offerors  and  the  available  evaluation 
specialists;  however,  the  schedule  for  these  closing  days  must  be  consid¬ 
ered  carefully  by  the  SSAC  and  SSEB  planners. 

Scoring  of  Cost  was  a  major  departure  from  past  practices,  and,  in  previ¬ 
ous  SSEBs,  it  was  regarded  somewhat  as  a  self-scored  area,  already  offered 
in  proposals  in  quantifiable  form.  In  the  case  of  this  solicitation; 
however,  the  RFP  required  the  offerors  to  guarantee  DTC  and  O&S  Costs  and 
allowed  great  latitude  in  presenting  these  guarantees.  As  such,  the 
evaluation  involved  far  more  numerical  comparison  and  analysis  than  in  the 
past.  Inclusions,  exclusions,  limitations  and  financial  constraints  on 
contractor  risk  required  that  the  SSEB’s  analysis  of  guarantees  consider 
many  strengths  and  weaknesses  which,  otherwise,  would  not  be  apparent.  In 
retrospect,  scoring  Cost  was  not  just  desirable,  it  was  a  necessity. 

There  is  every  reason  to  believe  that  future  SSEBs  should  be  prepared  to 
do  the  same.  However,  the  offerors  should  be  advised  in  the  RFP  that 
coverage,  including  depth  and  firmness  of  guarantees,  warranties,  and 
liabilities  beyond  just  definitions,  could  be  a  plus  in  the  scoring.  The 
suggestion  is  based  on  this  board's  experience  with  both  DTC  and  O&S, 
wherein  the  Evaluation  Plan  allowed  for  considerable  weight  to  be  assigned 
against  non-quant i f iable  or  ha rd-to-quanti fy  features  of  these  guarantees. 

For  example,  one-fifth  of  the  Procurement  (DTC)  Element  score  was  allotted 
to  other  procurement  costs,  but  because  theie  was  no  specific  cost  that 
could  be  associated  with  this  topic,  which  was  dominated  by  spare  parts 
costs,  the  allocation  probably  was  too  high. 

It  also  became  clear  during  the  evaluation  that  more  guidance  should  have 
been  provided  in  the  RFP  to  encourage  flexibility  in  formulating  guaran¬ 
tees.  A  majoi"  lesson  learned  in  the  new  approach  is  that  if  the  govern- 
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ment  has  a  minimum  standard  for  coverage  which  will  be  used  in  the 
evaluation,  it  should  be  stated  directly  in  the  RFP.  Stating  a  simple 
standard,  such  as  pounds  of  fuel  per  gallon,  would  have  avoided  inconsis¬ 
tencies  in  O&S  costs  and  saved  unnecessarily  repeated  discussions  and 
reevaluation .  In  another  case,  not  nearly  so  simple,  the  production  sched¬ 
ule  provided  with  the  RFP  tended  to  emphasize  one  level  of  delivery,  and 
the  importance  of  flexibility  (driven  by  availability  of  future  procure¬ 
ment  funds)  was  not  emphasized  enough.  Clearly,  the  offerors  and  the 
evaluators  realized  beforehand  that  production  schedules  made  years  in 
advance  of  obligation  authority  will  change,  and  some  range  of  flexibility 
should  have  been  expressed  in  the  RFP  without  leaving  the  impression  that 
there  was  a  single,  set  schedule.  The  needed  flexibility,  therefore,  was 
clarified  through  the  EOC  process  and  the  follow-on  discussions,  and  all 
offerors  met  the  requirement.  Time  and  effort  also  could  have  been  saved 
if  economic  escalation  methodology  had  been  defined  in  the  RFP,  along  with 
midpoint  expenditures  for  inflation.  While  the  economists  assigned  to 
this  board  had  no  specific  difficulty  in  handling  inflation  indicies, 
future  SSEBs  might  want  to  establish  a  uniform  definition.  For  the  T800, 
each  offeror  defined  his  own  methodology,  and  it  was  necessary  for  the 
evaluators  to  develop  three  separate  assessments  of  the  methodology  used. 
Another  DTC  concern  was  that  much  attention  was  placed  on  the  two  quanti¬ 
ties  specified  in  the  RFP  (250  units  and  10,000  units),  and  the  desire  for 
previously  mentioned  flexibility  was  not  clear.  It  should  be  emphasized 
in  future  RFPs  that  benchmarks  be  used  only  in  the  context  that  many  other 
production  points  are  to  be  evaluated.  The  important  issues  here  should 
be  the  total  commitment  to  DTC  with  a  wide  array  of  potential  quantities. 

An  0&S  goal  for  a  set  number  of  engine  operating  hours  was  found  to  be 
desirable.  However,  this  SSEB  found  that  the  goal  translated  into  overly 
restrictive  guidance  that  could  have  limited  some  of  the  creative  and 
attractive  approaches.  As  the  date  for  BAFO  approached,  however,  the 
offerors  continued  to  refine  their  bids,  probably  because  of  a  recognized 
need  to  provide  depth  and  firmness  in  the  guarantees  and  liabilities  that 
they  were  willing  to  accept. 

In  the  final  analysis,  the  new  and  innovative  approach  greatly  improved 
the  acquisition  process  and  resulted  in  contractually-binding  guarantees 
for  DTC  and  0&S.  These  guarantees  extend  for  up  to  20  years.  Should  they 
cease  to  exist  before  the  full  life,  they  will  have  been  in  place  long 
enough  to  drive  home  the  point  that  contractors  are  expected  to  share 
responsibility  for  0&S  risks  along  with  the  government.  This  shift, 
indeed,  is  a  new  way  of  doing  business,  and  the  government  now  has  the 
contractual  tools  in  place.  If  the  Army,  DOD,  and  Congress  do  not  hold 
the  contractor  to  the  conditions  of  these  guarantees  or  if  the  program  is 
allowed  to  vacillate  and  change  continuously,  the  advantages,  likewise,  may 
not  hold.  Congress,  DOD  and  the  Army  must  continue  to  be  a  wise  spender 
of  the  tax  dollar. 

F.  CONTRACTING/LEGAL 

The  core  Contracts/Legal  team  provided  support  for  all  areas  involved  in 
the  evaluation.  One  specialist,  was  assigned  to  each  proposal  (offeror) 
and  was  responsible  for  all  contracting  actions  and  contact  with  that 
offeror.  In  addition,  another  specialist  was  assigned  responsibility  for 
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preaward  surveys,  subcontracting  plans,  RFP  amendments,  and  audits. 

During  each  step  in  the  evaluation  process,  it  was  essential  that  at  least 
one  legal  advisor  be  available  to  the  Contracting  Officer  and  the  leader¬ 
ship  of  the  SSEB.  In  addition,  Contracting  and  Legal  representation  was 
mandatory  at  all  meetings  with  the  offerors.  Careful  and  thorough  coordi¬ 
nation  was  exercised  by  the  team  to  ensure  that  information  disseminated 
to  the  offerors  was  consistent  and  that  a  fair,  competitive  environment 
was  maintained.  Administratively,  the  Contracting/Legal  staff  was  critical 
to  the  EOC  process  and  was  responsible  for  developing,  controlling,  and 
maintaining  all  necessary  files. 

During  the  period  in  which  parallel  discussions  were  held  with  the 
offerors,  the  sudden  increase  in  workload  required  that  two  additional 
legal  advisors  be  assigned.  Without  question,  this  addition  to  the  staff 
left  the  sponsoring  command  shorthanded  for  several  weeks  and  probably 
placed  a  severe  hardship  on  the  parent  organization.  In  future  SSEBs ,  it 
is  highly  recommended  that  this  new  way  of  doing  business  be  accounted  for 
early  and  that  schedules  which  force  parallel  meetings  be  avoided,  if  at 
all  possible. 

The  task  before  this  team  was  to  implement  the  acquisition  strategy  by 
focusing  on: 

o  maximizing  competition  during  all  phases  and  at  all  levels  of  the 
program; 

o  providing  offerors  the  opportunity  to  propose  and  to  accomplish  the 
program  they  believed  would  best  satisfy  the  Army's  requirements;  and 

o  shifting  a  large  share  of  the  program  risk  from  the  government  to 
the  contractor. 

The  contractors'  and  Army's  commitments  to  competition  have  been  made  at 
the  outset,  beginning  with  development  and  continuing  through  production. 
This  competition  will  force  contractors  to  pursue  design  and  development 
vigorously  and  activate  the  required  organizations  to  implement  the  RAM, 
ILS,  and  Production  Competition  requirements.  To  obtain  long-term  bene¬ 
fits  and  maintain  a  competitive  environment  with  the  two  developer  teams, 
offerors  were  required  to  guarantee  their  cost  and  performance  now,  rather 
than  cite  future  goals  to  be  negotiated  later.  For  a  summary  of  the 
long-term  benefits  resulting  from  this  new  way  of  doing  business,  refer  to 
T800  New  Initiatives,  Figure  1,  of  this  report.  It  will  be  noted  in  the 
Competition  Area  that  each  of  the  developer  teams  is  contractually-bound 
to  establish  and  maintain  a  minimum  of  two  sources  for  all  parts.  For  the 
first  time  in  Army  aviation,  the  selection  process  forced  the  offerors  to 
address  such  issues  in  their  proposals  and  to  begin  execution  of  their 
related  plans  early  in  development. 

In  another  first,  this  SSEB  faced  three  different  teaming  arrangements  -  a 
joint  venture,  a  leader/follower  and  a  partnership.  All  were  viable 
arrangements,  devised  specifically  to  meet  the  Army's  T800  Engine  require¬ 
ments  for  end  item  competition.  Although  each  of  these  teams  was  char- 
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tered  differently,  they  presented  similar  problems,  which  included 
detailed  development  and  transfer  of  technology  and  data,  the  basis  in  law 
and  in  corporate  rules  which  authorize  the  teaming  arrangement,  and 
establishment  of  an  enforceable  contract  with  the  team.  Establishment  and 
maintenance  of  dual  end  item  sources  required  comprehensive  procedures  for 
data  and  technology  transfer,  particularly  as  related  to  Class  I  and 
II  Engineering  Change  Proposal  (ECP)  data  and  to  Material  Review  Board 
(MRB)  actions.  Although  MRB  actions  do  not  individually  impact  form,  fit 
and  function,  they  might  accumulate  with  time  and  adversely  impact  inter¬ 
changeability.  To  guard  against  that  concern,  procedures  developed  early 
in  discussions  will  preclude  a  sole-source  situation,  and  the  government 
now  has  the  right  to  direct  the  transfer  and  to  develop  a  method  of 
pricing  with  ultimate  resolution  in  the  disputes  procedures. 

An  early  concern  addressed  whether  the  established  teaming  arrangements 
complied  with  state  law  and  with  the  rules  of  each  corporation.  Documen¬ 
tation  was  submitted  with  the  proposal  as  substantiating  data;  however, 
the  government  needed  assurance  that  a  contract  not  be  signed  with  a  shell 
organization  that  had  no  assets  and  no  employees,  thereby  rendering  the 
contract  non-enforceable .  Therefore,  it  was  necessary  to  incorporate 
joint/several  liability,  whereby  the  corporations  which  formed  the  team 
would  be  signatories  to  the  agreements  and  responsibilities  established  by 
the  entity.  Care  was  necessary  to  assure  that  the  individuals  who  signed 
the  contract  were  authorized  to  bind  the  corporation  and  the  team.  In  the 
process,  the  subject  of  anti-trust  concerns  also  was  addressed,  especially 
as  related  to  the  follow-on,  competitive  production  phases  of  the  program. 
It  became  the  contractor's  responsibility  to  assure  that  antitrust  provi¬ 
sions  were  properly  considered  and  not  violated. 

As  discussed  earlier  in  this  report,  flexibility  to  define  program  details 
was  given  to  the  offerors.  Contractually-binding  plans  were  developed 
which  describe  how  the  offeror  will  fulfill  the  requirements.  These  plans 
also  will  be  used  by  the  Army  to  track  progress,  manage  the  programs,  and 
provide  a  basis  for  down-select  which  is  scheduled  to  occur  after  3  years 
of  development.  To  reiterate,  the  program  approach  and  milestones  unique 
to  each  contract  have  been  established  in  program  plans  at  the  outset,  and 
each  team's  progress  will  be  measured  against  the  deliveries  stated  in 
those  plans.  It  is  important  to  note  that  for  evaluation  purposes,  the 
Systems  Specification  had  precedence  over  the  program  plans,  as  indicated 
in  the  RFP.  However,  it  was  mandatory  to  have  the  offerors  contractually 
commit  to  those  program  plans. 

Obviously,  any  inconsistency  or  discrepancy  between  the  program  plans  and 
the  System  Specification  presented  a  problem.  The  solution  was  to  deter¬ 
mine  whether  the  System  Specification  met  the  RFP  requirements  before 
attempting  to  evaluate  contractually-binding  program  plans.  With  a 
positive  finding  qri  that  question,  evaluation  scores  then  were  applied  to 
the  program  plans  for  performing  against  the  System  Specification.  In 
future  acquisitions,  it  is  recommended  that  early  steps  be  taken  to 
include  clear  instructions  that  the  System  Specification  takes  precedence. 


In  summary,  shifting  the  risk  to  the  contractors,  was  achieved  successful¬ 
ly.  Cost  risk  was  accepted  by  the  contractors,  beginning  with  the  firm 
fixed  price  FSD  contract.  They  are  committed  to  demonstrate  the  govern¬ 
ment's  requirements  during  FSD,  including  demonstration  of  the  RAM  re¬ 
quirements.  The  contracts  also  established  that  the  government  will  not 
pay  directly  for  facilitization ,  including  brick  and  mortar,  and  all 
tooling.  In  that  vein,  the  contractor  will  assume  the  risk  of 
facilitizing  in  the  time  frame  necessary  to  support  program  requirements, 
and  it  is  expected  that  they  can  recoup  these  costs  on  an  indirect  basis 
in  some  later  phase  of  the  program.  It  must  also  be  noted  that  the 
contractors  have  assumed  a  high  level  of  program  risk  by  guaranteeing  a 
ceiling  price  for  the  end  item  and  a  maximum  cost  to  the  government  for 
operating  and  support  of  the  T800  engine.  Included  are  substantial 
contractor  liabilities  if  these  guarantees  are  not  met. 

In  the  foregoing  discussions,  a  point  was  made  that  this  acquisition  has 
shifted  risk  to  the  contractors.  Another  point  was  that  the  major  differ¬ 
ence  between  this  acquisition  and  others  is  that  requirements  were  estab¬ 
lished  for  contracting  purposes  in  lieu  of  objectives  or  goals  th  ly 
have  dominated  previous  programs.  For  the  T800,  the  contractors 
accomplish  all  that  is  necessary,  including  redesign,  retest, 
requalification  and  retrofit,  at  no  increase  in  contract  price,  to  demon¬ 
strate  the  FSD  requirements. 
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SECTION  V 


OFFERORS'  RESPONSES  TO  NEW  INITIATIVES  OF  RFP 


Because  the  T800  Engine  RFP  and  its  innovations  were  such  a  departure  from 
past  practices,  both  the  SSAC  and  SSEB  believed  that  candid  comments  from 
industry  would  be  beneficial  to  assessing  how  well  the  process  met  the 
Army's  objectives.  With  that  desire  in  mind,  letters  from  the  Commander, 
U.S.  Army  Materiel  Command,  were  sent  to  each  offeror's  corporate  leader¬ 
ship  to  solicit  their  views.  Responses  were  received  prior  to  the  final 
SSAC  briefing,  but  they  were  safeguarded  at  AMC  until  after  the  SSA 
briefing  so  as  to  not  reveal  contents  to  members  of  the  SSAC  or  SSEB  until 
the  selection  process  was  completed.  Overall,  the  otferors  believed  that 
the  that  new  approach  to  acquisition  was  a  positive  step  forward  for  the 
Army.  More  specific  comments,  as  related  to  the  major  topics  covered  in 
the  process,  are  given  in  the  paragraphs  that  follow. 

A.  REQUEST  FOR  PROPOSAL 

Each  offeror  recognized  the  performance-oriented  RFP  as  an  improvement  and 
a  welcome  departure  from  past  practices.  One  offeror  noted,  however,  that 
"inclusion  of  many  military  specifications  by  reference  ran  counter  to  the 
idea  of  a  short,  performance-oriented  RFP  document."  Another  believed 
that  there  was  'little  opportunity  for  the  responder  to  use  his  own 
initiative  in  establishing  favorable  trade-offs  between  engine  design 
parameters;  whereas,  a  true  performance-oriented  program  would  have 
specified  some  gross  overall  parameters  and  left  the  responder  to  deter¬ 
mine  how  best  to  meet  the  requirements'.  On  the  other  hand,  other 
offerors  indicated  that  they,  themselves,  had  a  problem  breaking  the 
business  as  usual  attitude  and  did  not  take  full  advantage  of  latitude 
allowed  them  in  the  RFP.  It  was  pointed  out  that  the  Army  contributed  to 
that  interpretation  with  the  original  "standard"  T800  RFP  that  was  sent  to 
industry  (August  1984)  for  review  with  the  many  specifications  included. 
(Editor's  note:  Several  follow-on  refinements  without  the  large  number  of 
specifications  also  were  reviewed  by  the  industry.  The  last  of  these 
preliminary  editions  was  very  close  to  the  final  version  issued  on  5 
December  1984.)  One  offeror  believed  that  the  2,000-page  limit  on  re¬ 
sponses  was  good  and  that  it  ’forced  a  more  tightly  constructed  and  well 
thought-out  response.'  Relative  to  the  90-day  response  time  between  the 
RFP  issue  and  the  due  date  for  proposals,  the  offerors  believed  that  they 
had  adequate  preparation  time.  Overall,  the  RFP  was  regarded  positively, 
and  the  tone  was  that  once  the  Army  and  the  industry  become  more  accus¬ 
tomed  to  the  new  approach,  the  benefits  to  both  parties  will  increase 
proport lona 1 ly . 

B.  ERRORS,  OMISSIONS,  AND  CLARIFICATIONS  (EOC)  PROCESS 

All  offerors  recognized  the  need  for  the  EOC  process.  It  permitted  them  an 
opportunity  to  fill  voids,  develop  a  better  understanding  of  RFP  require¬ 
ments,  and  adjust  their  proposals  accordingly.  Some  offerors  commented 


that  EOC  responses  might  not  have  been  delivered  to  the  questioners  fast 
enough,  nor  disseminated  to  all  interested  members  of  the  SSEB.  One 
offeror  pointed  out  that  an  EOC  often  required  more  than  the  allotted 
3  days  to  develop  a  complete  answer.  However,  the  overall  reaction  to  the 
process  was  positive,  and  they  believed  that  it  should  be  continued.  In 
terms  of  value,  all  offerors  found  the  face-to-face  discussions  and 
negotiations  to  be  more  beneficial  than  EOCs. 

C.  DISCUSSIONS  AND  NEGOTIATIONS 

All  offerors  commented  that  the  fact-finding  discussions  and  negotiations 
provided  the  best  means  for  clearing  up  their  proposals  and  for  under¬ 
standing  the  government’s  requirements.  Most  offerors  commented  that  the 
SSEB  negotiated  the  desired  government  position  in  a  more  complete  manner 
than  has  been  encountered  in  previous  procurements.  One  offeror  felt 
strongly  that  it  was  in  the  best  interest  of  the  Army  and  the  contractor 
to  use  negotiations  as  an  allowable  approach  to  the  solution  of  a  problem. 
Further,  it  was  pointed  out  that,  ' pre-determined  courses  of  action  (that 
leave  no  room  for  negotiations)  are  only  as  good  as  our  joint  ability  to 
anticipate  future  events.'  It  was  noted  that  the  discussions  were  more 
complete  that  those  encountered  in  past  procurements  and  were  extremely 
helpful.  Overall,  the  use  of  open  and  in-depth  negotiations  was  found  to 
be  indispensable  to  the  success  of  a  performance-oriented  source  selection 
process . 

D.  BEST  AND  FINAL  OFFER 

Some  offerors  noted  that  the  uniqueness  of  this  source  selection  process 
and  its  schedule  did  not  place  a  well  defined  split  between  fact  finding 
and  negotiations  before  the  BAFO.  Otherwise,  there  was  little  in  the  way 
of  comments  on  the  BAFO.  The  general  feeling  seemed  to  be  that  discus¬ 
sions  were  held  in  a  reasonable,  business  manner,  and,  as  one  offeror 
stated,  'not  in  the  much  maligned  "BAFO"  approach.'  Further  it  was 
pointed  out  'hat  ’this  process  affords  the  contractor  the  opportunity  to 
continue  to  improve  his  proposal  right  through  submission  of  the  BAFO. * 

The  consensus  was  that  all  offerors'  BAFOs  presented  ..he  Army  with  ex¬ 
tremely  attractive  alternatives,  and  a  difficult  source  selection.  One 
offeror  stated  that  the  BAFO  response  consisted  of  the  model  contract  and 
a  large  number  of  plans,  and  it  would  have  been  helpful  if  this  intention 
had  been  identified  in  the  RFP .  Overall,  however  offerors  believed  that 
the  Army  has  developed  a  much  better  approach. 

E.  CONDUCT  OF  BOARD 

All  offerors  commented  that  the  high  level  of  professionalism  of  the  SSEB 
was  a  key  factor  in  succeeding  with  the  new  and  innovative  approach.  There 
were  a  few  comments  on  the  apparent  lack  of  time  available  for  proper  or 
complete  dissemination  of  contractor  supplied  information  to  the  SSEB; 
however,  as  one  offeror  noted,  this  source  selection  was,  'a  quantum  step 
forward  in  the  acquisition  process.'  Another  offeror  tempered  his  enthu¬ 
siasm  with,  this  new  approach  for  enhanced  competition  and  strong  commit¬ 
ments  from  Industry  is  in  the  right  direction,  but  care  must  be  exercised 
by  the  government  not  to  overshoot  the  mark.'  Further,  another  noted, 


the  government  must  give  thought  to  a  sufficiently  large  production  base 
for  industry's  shareholders  to  achieve  a  reasonable  return  on  their 
investment.'  The  SSEB  activity  was  summarized  by  one  offeror's  comment 
as,  'an  excellent  model  for  future  acquisi t 1 ons . ' 
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SECTION  VI 


RECOMMENDATIONS/ LESSONS 


During  the  process  of  developing  the  RFP,  structuring  the  SSEB,  and 
conducting  the  evaluation,  many  firsts  were  encountered.  Some  of  them  led 
to  a  need  to  change  course,  and  others  proved  to  be  the  basis  for  lessons 
learned.  All  ol  these  details  are  a  reflection  of  the  Board’s  evaluation 
process.  It  has  been  the  attempt  in  this  after-action  report  to  surface 
those  topics  believed  to  be  of  most  interest  for  future  board  applica¬ 
tions.  To  assist  the  reader,  this  section  lists  recommendations  made  by 
the  Board  and,  for  reference,  indicates  sections  where  further  discussion 
mav  be  found. 


1.  RFP  Coordination 

a.  Coer  1 :  r.a  t  g  -MT  -wi  th  government  and  industry. 
Hold  ore-proposal  conferences  to  communicate  the 
government's  intentions.  Develop  understanding 
with  potential  offerors.  Describe  new  initiatives 
and  expected  departures  from  past  practices. 


Summary 
Section  I-A 
Section  I-B 
Section  1-D 


b.  Provide  early  opportunity  for  detailed 
learning  discussions  with  offerors  on  proposed 
plans,  guarantees,  and  long-range  commitments. 

2 .  Content  of  RFP 

a.  Exercise  caution  in  eliminating  government's 
tools  for  measuring  contractor  performance  (CDRLs). 
Even  so,  nearly  half  can  be  eliminated  from  standard 
RFPs  . 

b.  Provide  Federal  Acquisition  Regulation,  Army 
Regulation,  and  DARCOM  Pamphlet  on  the  source 
selection  process  to  offerors  with  the  RFP.  Also 
provide  portions  of  the  Evaluation  Plan. 

c.  Identify  in  the  RFP  the  kind  of  substanti¬ 
ation  required  in  the  offeror's  test  program  plans. 
Include  sufficient  detail  to  define  pass-fail 
criteria . 


Section  I-A. U 


Section  I-A. 3 


Section  II/Intro 
Section  III-C 


Section  IV-C 


d.  Provide  weighting  of  areas,  elements,  and 
factors  to  be  used  in  scoring  of  proposals 
(published  in  RFP).  Use  these  weights  to  emphasize 
importance  attached  to  each  new  initiative  in  the  RFP. 


Section  I-A 
Section  1 1 - G 


Section  IV-E 


e.  Require  offerors  to  submit  RDT&E  cost  data 
on  computer  floppy  disks,  using  LOTUS  1,  2,  3  soft¬ 
ware.  Also  request  that  hard  copies  be  provided. 

f.  Provide  guidance  to  encourage  offerors  to 
be  flexible  in  formulating  long-range  guarantees. 
Provide  a  minimum  standard  as  the  baseline. 

g.  Emphasize  scoring  of  the  Cost  Area. 

h.  Consider  a  cost  evaluation  structure  that 
emphasizes  only  quantifiable  features,  not  just 
definitions,  but  depth  and  firmness  of  guarantees, 
warranties,  and  liabilities. 

i.  Provide  specific  parameters  when  appropriate, 
rather  than  allowing  each  offeror  to  calculate 
standard  values. 

j.  Emphasize  the  need  for  broad  coverage  in 
guarantees  for  procurement,  considering  that 
production  schedules  and  lot  sizes  will  be  driven 
by  availability  of  funds  in  the  out-years. 

k.  Define  economic  escalation  methodology. 

l.  Assign  benchmarks  for  operating  hours  to  be 
used  as  guidance  in  developing  operating  and  support 
cost  guarantees. 

m.  Define  requirements  for  interchangeability, 
regardless  of  the  number  of  vendor  sources  involved 
in  the  competitive  procurement. 

n.  Define  whether  facilitization  will  be  funded. 


o.  Require  that  offerors  develop  contractual 
plans  for  priced  options  as  related  to  technical 
data  packages  (TDPs)  and  qualification  of  alternate 
vendors . 

p.  Establish  that  a  contractual  plan  will  be 
required  to  pursue  enhancement  of  small,  small 
disadvantaged ,  and  women  owned  businesses. 

3.  Staffing  of  SSEB 


Section  IV-E 

Section  IV-E 
Section  IV-E 

Section  IV-E 

Section  IV-E 

Section  IV-E 
Section  IV-E 

Section  IV-B 

Section  IV-B 
Section  IV-F 

Section  IV-B 

Section  IV-B 


a.  Assign  key  personnel  early  enough  to  play 
significant  roles  in  the  preparation  of  the  RFP. 


Section  I-B 


Section  I -B 


b.  Assign  team  leaders  (Chairman,  Deputy,  Chiefs, 
other  key  positions)  6  months  prior  to  release  of  the 
RFP.  Require  full-time  service  of  chairman  and/or 
deputy  2  months  before  releasing  RFP. 

c.  Require  team  leaders  to  define  skills  needed 
for  the  evaluation  with  enough  lead  time  to  identify 
and  assign  candidates  at  least  30  days  before  pro¬ 
posals  are  received. 

d.  Develop  AMC/AVSCOM  policy  to  establish  appro¬ 
priate  means  for  making  assignments  to  SSEBs  less 
painful  to  parent  organization. 

e.  Assign  dedicated  personnel  for  controlling 
EOC  process. 

f.  Provide  specialist  for  orchestrating  formal 
and  informal  presentations. 

g.  Provide  special  training  for  clerical  person¬ 
nel  for  operation  of  new  equipment  (word  processing, 
computer) . 

h.  Provide  economists  for  evaluation  of  long- 
range  cost  aspects  related  to  procurement  and 
operating  support  costs. 

4.  Training  of  Ivlmtora 

a.  Begin  training  of  evaluators  1  week  before 
proposals  are  received.  Cover  the  evaluation  tech¬ 
niques  necessary  to  address  new  initiatives. 

b.  Develop  samples  of  completed  evaluation 
sheets  to  be  available  during  training  period. 

c.  Define  use  of  the  Error,  Omission,  and 
Clarification  (EOC)  process,  (along  with  respon¬ 
sibilities  of  evaluators,  chiefs,  and  Contracting 
Officer)  during  the  training  period,  emphasizing 
the  purpose  of  the  EOC  form. 

d.  Conduct  on-site  briefing  at  least  2  days 
before  proposals  are  received  to  develop  uniform 
understanding  across  all  segments  of  the  SSEB. 

As  a  minimum,  cover  background  of  RFP,  acqui¬ 
sition  strategy,  and  rules  of  conduct. 

e.  Require  that  evaluators  thoroughly  read 
and  understand  the  RFP  and  Board  Operating  Proce¬ 
dure  (BOP)  documents  during  the  training  period. 


Section  I-B 

Section  I-B 

Section  I-B 

Section  I-B 

Section  I-B 

Section  IV-E 

Summa ry 
Section  I-E 

Section  I/Intro 

Section  I-B 

Section  I/Intro 

Section  I/Intro 


Section  I-C 


f.  Establish  a  BOP  to  describe  plans  for  the 
rating  systems  to  be  used  for  performance  appraisals 
of  civilian  ■  r ■'  ...^luary  personnel. 

5 .  Overview  by  Offerors 

a.  Request  overview  briefing  by  offerors  to 
SSEB  within  2  days  after  proposals  are  received. 

b.  Request  a  short  video  tape  (15-20  minutes) 
from  the  offerors  to  be  delivered  with  the  proposals. 

6.  Errors,  Omissions,  and  Clarifications  (EOCs) 

a.  Devote  the  first  week  to  10  days  only  to 
reading  and  understanding  the  offeror's  documents  and 
tapes  to  develop  the  basis  for  the  EOC  process. 

b.  Ensure  that  the  evaluation  team  has  completed 
its  internal  coordination  before  generating  detailed 
questions  for  EOCs. 

c.  Review  proposed  EOCs  thoroughly  before  issuing 
any  to  the  offeror. 

d.  Conduct  daily  staff  meetings  with  key  individ¬ 
uals  to  ensure  that  the  coordination  has  taken  place 
with  all  principals. 

e.  Limit  the  EOC  process  to  a  period  of  about 

2  weeks  to  ensure  that  it  does  not  grow  beyond  its 
purpose . 

7 .  Discussions  and  Negotiations 

a.  Concentrate  effort  on  the  fact-finding  sessions 
to  develop  clarification  of  proposal  details  needed  for 
evaluation. 

b.  Establish  an  agenda  for  fact-finding 
discussions  and  provide  it  to  the  offerors  at  least 

3  days  in  advance. 

c.  ControL  alL  contacts  with  the  offeror  and 
establish  that  all  meetings  are  to  be  conducted 
by  the  Contracting  Officer. 

d.  Provide  minutes  to  the  offerors  within 
3  working  days  following  fact-finding  discus¬ 
sions  or  negotiations . 


Section  II-A 


Section  II-A 
Section  II-B 


Section  II-A 


Section  II-A 


Section  II-B 


Section  II-A 


Section  II-B 


Section  II-C 


Section  II-C 


Section  II-C 


Section  II-C 


e.  Do  not  begin  fact-finding  discussions  until 
the  EOC  process  is  completed  (if  schedule  permits). 
All  offerors  commented  that  fact  finding  and 
negotiations  were  the  best  means  for  clearing  up 
their  proposals  and  for  understanding  the  govern¬ 
ment's  requirements. 

f.  Avoid  parallel  meetings  with  offerors 
(meetings  in  different  locations  at  the  same 
time ) . 

g.  Assess  whether  follow-up  negotiations 
with  winners  may  be  beneficial. 

h.  Provide  at  least  one  legal  advisor  to 
support  the  SSEB. 

i.  Ensure  that  the  technology  transfer 
between  partners  is  sufficient  to  prevent  one 
partner  from  becoming  a  sole-source  supplier. 

j.  Ensure  that  offerors  understand  that 
the  government  intends  for  the  offerors  to  be 
jointly  liable  via  their  partnership  and 
individually  liable  via  their  roles  as  competing 
end  item  suppliers. 

k.  Require  assurance  that  a  developer 
team's  business  arrangement  considers  all  anti¬ 
trust  provisions  in  preparing  their  proposals 
to  the  government. 

l.  Ensure  that  acquisition  strategy  and 
milestones  defined  in  the  RFP  are  available 
for  potential  down-stream  competition. 

8.  Evaluation  Structure  and  Strategy 


Section  II-C 
Section  V-C 


Section  1-A 
Section  IV-E 
Section  1V-F 

Section  IV/Intro 


Section  IV-F 


Section  IV-F 


Section  IV-F 


Section  IV-F 


Section  IV-E 


a.  Include  Survivability/Vulnerability  in 
appropriate  element  of  Technical  Area  instead 
of  RAM/ ILS . 

b.  Require  offerors  to  cross  reference 
the  various  volumes  of  the  proposal  so  that 
evaluators  will  not  miss  necessary  information. 

c.  Give  MANPRINT  more  visibility  in  the 
RAM/ ILS  structure,  and  increase  the  weighting 
of  Quality  Engineering. 


Section  IV-C 


Section  IV-D 


Section  IV-D 


d.  Provide  an  SSEB  structure  that  will 
strengthen  and  integrate  MANPRINT  with  primary 
elements  of  RAM.  Tie  Human  Factors  to  MANPRINT. 


Section  IV-D 
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9.  Best  and  Final  Offer  (BAFO) 


a.  Provide  offeror  at  least  2  weeks  to  pre¬ 
pare  the  BAFO. 


Summary 


b.  Provide  model  contracts  to  offerors  at 
least  2  weeks  prior  to  BAFO  due  date. 

c.  Define  the  period  of  time  that  the  BAFO 
must  remain  valid  (such  as  90  days). 

d.  Allow  at  least  2  weeks  between  receipt 
of  BAFO  and  the  completion  of  the  evaluation. 

10 .  Source  Selection  Advisory  Council  (SSAC) 
Meetings 


Summary 


Section  II-F 


Section  1V-E 


a.  Conduct  frequent,  separate  meetings  with 
SSAC  members  to  enhance  the  flow  of  information 
during  follow-on  formal  briefings  to  the  SSAC. 

b.  Schedule  SSAC  meetings  during  the  eval¬ 
uation  roughly  a  month  apart.  They  are  an 
essential  ingredient  for  a  successful  evaluation. 


Summary 
Section  II-H 


Section  I/Intro 


11.  Board  Report 


a.  Keep  running  logs  of  lessons  learned 
to  be  incorporated  in  reports. 

b.  Document  all  steps  taken. 


Summary 


Section  Ill/Intro 


c.  Disseminate  the  After-Action  Report 
to  offerors  as  well  as  to  government 
organizations . 


Section  Ill/Intro 
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The  purpose  of  this  report  Is  to  chronicle  the  lessons  learned  during  th< 


MSE  Source  Selection.  It  Is  intended  to  provide  help  to  those  who  will  folloi 
the  MSE  NDI  acquisition  strategy  in  the  future.  It  is  not  intended  to  be  criti 
cal  of  any  organization  or  person  who  participated  in  the  MSE  evaluation,  hin 
sight  is  always  perfect. 

The  MSE  source  selection  began  with  the  receipt  of  proposals  on  l  OCT  198' 
and  ended  in  AUG  1985  with  the  final  briefing  to  the  Senior  Advisory  Counci 
(SAC).  During  this  period  thousands  of  pages  of  proposals  were  evaluated,  thre 
demonstrations  conducted  overseas,  hundreds  of  hours  spent  in  negotiations,  nup 
erous  briefings  given  to  all  levels  of  the  Army,  six  model  contracts  prepared 
and  a  final  report  written.  It  is  an  effort  that  all  who  participated  in  can  b 
Justly  proud  of. 


This  report  will  not  dwell  on  the  administrative  procedures  required  to  ru 
a  source  selection  evaluation,  Instead  it  will  concentrate  on  the  unique  aspect 
of  the  MSE  acquisition  strategy.  This  report  Is  Intended  to  be  a  ready  refer 
ence  to  help  others  in  conducting  an  NDI  source  selection. 

This  report  is  structured  in  chronological  order  beginning  with  th 
generation  of  the  requirement  and  ending  with  the  source  selection. 


MSE  RFP  AND  EVALUATION  HISTORY 


The  initial  MSE  RFP  prepared  in  early  1983  was  for  a  division  mobile 
subscriber  system  which  would  augment  the  planned  TRI-TAC  C3  automatic  wire  sub¬ 
scriber  system.  Based  on  the  recommendations  of  the  Battlefield  Communications 
Review  (BCR)  held  in  the  Fall  of  1983,  the  Army  redirected  the  program  to  pro¬ 
vide  a  secure  automatic  mobile  subscriber  system  replacing  the  current  and 
planned  C3  systems  throughout  the  Corps  area.  The  Under  Secretary  of  the  Army 
approved  the  program  and  directed  that  the  Army  procure  an  available  system  on  a 
nondevelopmental  basis.  The  RFP  required  that  the  offerors  propose  only  total 
systems  (hardware  and  software)  that  had  been  designed,  developed,  tested  and 
whose  principal  components  were  currently  in  production.  The  offer  was  to 
include  a  system  performance  specification  describing  the  performance  character¬ 
istics  of  the  proposed  system  which  would  satisfy  the  general  requirements 
presented  in  the  RFP.  The  usual  MIL-Standard  requirements  were  included  in  the 
RFP  as  "desired  features".  It  was  recognized  that  the  NDI  approach  would  pro¬ 
vide  a  system  that  did  not  meet  all  the  desired  features.  However,  the  Army  was 
prepared  to  accept  less  than  a  "full  MIL  System"  to  gain  the  cost  and  schedule 
advantages  of  an  NDI  System. 

The  RFP  also  required  a  demonstration  of  the  major  system  characteristics 
using  production  equipment  in  its  final  offered  configuration. 

The  Army  did  not  impose  any  of  the  logistics  and  management  controls  nor- 
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malLy  used  on  CE  programs.  Rather,  the  contractors  were  asked  to  propose  a 
system/equipment  acceptance  program,  contractor  training  (operator  and  mainte¬ 
nance),  contractor  materiel  fielding  and  a  proposed  configuration  management 
plan.  Several  other  requirements  were  added  to  insure  that  the  successful  con¬ 
tractor  continued  to  fulfill  his  contractual  obligations  during  the  life  of  the 
program.  These  included:  five  Firm  Fixed  Price  (FFP)  production  options,  FFP 
spares  for  the  life  of  the  program,  performance  warranties,  no  progress  pay¬ 
ments,  and  others. 


In  addition,  the  offeror  had  to  include  several  mandatory  priced  options 
which  would  be  delivered  with  the  original  fielding  and/or  all  systems  would  be 
retrofitted  at  some  appropriate  time  (all  FFP).  These  included  the  use  of  Army 
standard  support  equipment  (vehicles,  power  units,  trailers,  shelters,  etc.), 
provide  interfaces  to  Array  standard  and  NATO  CE  equipment  used  in  the  EAC,  and 
provide  high  altitude  EMP  protection. 


There  were  three  separate  contracts  Involved  in  this  program:  Basic,  Post 
Deployment  Software  Support  (PDSS),  and  the  Requirements  contract.  The  basic 
contract  was  to  be  a  firm,  fixed  price  procurement  vehicle  through  which  the 
Army  would  purchase  and  field  MSE  prime  equipment,  initial  spares,  ground  sup¬ 
port  equipment,  technical  data  and  warranty,  plus  provide  training.  The  PDSS 
contract  is  a  fixed  price  level  of  effort  vehicle  to  provide  all  necessary  soft¬ 
ware  support  throughout  the  life  of  the  equipment.  The  Integrated  Logistics 
Support  requirements  contract  is  a  "fixed  price"  contract  to  provide  all  neces¬ 
sary  maintenance  and  logistics  support  for  the  life  of  the  equipment. 
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The  procurement  mechanism  is  a  firm,  fixed  price  contract  by  which  the  Army 
plans  to  purchase  a  total  of  5  corps  and  26  divisions.  Because  of  budget  fluc¬ 
tuations,  the  offerors  priced  each  equipment  item  over  three  ranges  of 
quantities.  The  Army  requirements  were  placed  at  the  lower  end  of  range  B  with 
ranges  A  and  C  being  50%  less  and  50%  greater  respectively. 


The  RFP  was  released  on  2  July  1984.  Industry  was  briefed  prior  to  and 
subsequent  to  RFP  release.  The  purpose  of  these  briefings  was  to  clearly  define 
the  unusual  nature  of  this  RFP.  Several  amendments  resulted  prior  to  receipt  of 
proposals . 


On  12  September  1984,  the  MSE  Source  Selection/Evaluation  was  upgraded  to 
reflect  the  Importance,  complexity  and  financial  scope  of  this  major  acquisi¬ 
tion.  An  Executive  Committee  (EC)  was  established,  as  part  of  the  Evaluation 
Board,  made  up  of  the  EB  Chairman,  Deputy  Chairman,  Operations  Officer,  Deputy 
for  Systems,  Chairman  of  the  Standard  Evaluation  Board,  Project  Manager  Repre¬ 
sentative,  Legal  Counsel,  Contracting  Officer,  and  the  Panel  Chiefs.  The  EC 
reports  to  a  newly  established  Senior  Advisory  Council  (SAC  -  chaired  by  the  CG, 
Army  Materiel  Command)  which  replaced  the  former  Source  Selection  Advisory  Coun¬ 
cil  and  the  Secretary  of  the  Army  was  named  as  the  Source  Selection  Authority. 
The  Source  Selection  Evaluation  Board  (SSEB)  was  renamed  the  Evaluation  Board 
(EB)  with  members  and  duties  essentially  the  same  as  defined  in  the  original 
Source  Selection  Plan. 
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On  28  September  1984  the  Array  General  Counsel  provided  clarification  of  the 
acquisition  strategy  to  both  contractors.  This  clarification  stated  the  Army's 
desire  to  apply  commercial  practices  to  the  procurement  of  NDI  systems.  Fur¬ 
ther,  the  clarification  allowed  any  proposal  that  met  the  RFP  requirements  to  be 
accepted  and  authorized  submission  of  budgetary  cost  estimates  vice  cost  propo¬ 
sals.  Accordingly,  the  Army  defined  hardware  configurations  for  demonstration 
through  negotiations  rather  than  formalized  RFP  amendments  and  reproposal.  Each 
offeror  was  advised  that  since  the  proposals  could  be  different,  that  relative 
value  judgments  of  the  systems  would  be  subjective.  Finally,  the  Federal 
Acquisition  Regulations  (those  not  based  on  statute)  would  be  waived.  Proposals 
were  received  from  Rockwell/Collins  and  GTE  on  1  October  1984. 


The  Rockwell  proposal  also  included  an  alternate  proposal  with  ITT  as  a 
major  subcontractor.  It  should  be  noted  that  proposals  ranged  from  40,000  to 
60,000  single  spaced  pages.  Three  semi-trucks  were  required  to  transport  all 
documentation.  The  initial  evaluation  quickly  commenced  and  was  completed  in 
December  1984. 


Subsequently,  based  on  the  previous  guidance  from  the  General  Counsel's 
Office  the  acquisition  strategy  was  changed  to  require  the  bidders  to  propose  a 
"best  operational  system".  Potential  bidders  were  informed  that  variations  to 
the  RFP  would  be  evaluated  and  incorporated  if  determined  to  be  more  advanta¬ 
geous  to  the  Army.  In  defining  the  "best  operational  system"  for  each  offeror, 
the  Government  selected  a  system  from  alternatives  proposed  by  each  offeror 
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within  their  written  proposals  or  during  the  course  of  negotiations.  It  was 
continually  emphasized  that  the  final  system  configuration  was  the  offeror's  re¬ 
sponsibility  and  that  the  directed  system  was  selected  by  the  Army  without 
regard  to  price. 


To  facilitate  the  evaluation  process,  a  two  phase  source  selection  proce¬ 
dure  was  implemented.  The  Evaluation  Board  (EB)  completed  the  initial 
evaluation  following  the  formal  source  selection  procedures  and  documented  its 
findings  in  an  initial  evaluation  report.  Significant  Issues  were  surfaced  to 
the  EC.  The  EC  performed  an  early  and  Independent  review  and  identified  major 
deficiencies,  weaknesses,  and  high  risk  areas.  Based  on  this  review  and  the 
results  of  contractor  demonstrations  in  the  UK  (Dec  84  and  Apr  85)  and  France 
(Mar  85)  the  Initial  "best  operational"  configurations  were  negotiated  with  each 
contractor. 


Following  extensive  negotiations  with  each  contractor  and  SAC  approval, 
revised  proposals  describing  a  mutually  agreed  to  "best  operational  system"  were 
requested.  Proposals  were  received  and  evaluated  by  the  EB.  A  series  of  nego¬ 
tiations  followed  which  clarified  questionable  areas.  The  system  was  then 
frozen  and  final  validation  of  the  Statement  of  Work  and  specifications  were 
completed.  Model  contracts  were  prepared  by  the  EB  and  on  23  July  these  were 
forwarded  to  each  Contractor  with  a  request  for  a  "Best  and  Final  Offer" 
(BAFO).  The  BAFO  was  received  on  2  August  and  immediately  evaluated.  A  review 
of  the  responses  necessitated  reopening  negotiations  with  both  offerors  for  cla¬ 
rification.  Clarifications  to  the  BAFO  were  received  on  12  August.  The  SAC  and 
the  Under  Secretary  were  briefed  18-20  August  and  21  August  respectively. 
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To  summarise,  the  MSE  evaluation  employed  an  Innovative  procurement  stra¬ 
tegy  that  was  unique  in  acquisition  procedures  and  required  the  EB  to  accomplish 
the  following  major  tasks  not  normally  performed  by  evaluation  boards. 

a.  The  EB's  initial  definition  of  a  "Best  Operational  System"  incorporat¬ 
ing  specific  equipments  and  capabilities  from  each  offeror's  "menu"  provided 
flexibility,  but  placed  a  significant  burden  on  the  EB  to  fully  understand  the 
various  configurations  and  attempt  to  assess  the  impact  various  changes  would 
have  on  the  system  performance.  Although  each  offeror  was  informed  that  their 
Best  and  Final  Offer  was  entirely  their  own  proposal,  it  was  generally  recogniz¬ 
ed  that  the  Governments  "Best  Operational  System"  would  guide  the  contractor  in 
his  final  configuration. 

b.  Pursuant  to  the  initial  "Best  Operational  System,"  direct  negotiations 
by  Panel  Chiefs  with  each  contractor  resulted  in  a  mutually  acceptable  system 
that  was  generally  equal  in  terras  of  capability,  but  not  necessarily  equal  in 
numbers  of  equipments.  It  must  be  emphasized  that  this  system  was  negotiated 
without  the  benefit  of  price  data,  however,  negotiators  were  sensitive  to  possi¬ 
ble  allegations  of  "gold  plating." 

c.  The  preparation  of  a  Model  Contract  prior  to  BAFO  was  a  new  procedure. 
Normally  the  request  for  BAFO  is  done  by  letter  by  the  procurement  activity  and 
is  a  routine  action.  In  this  instance  the  EC  had  to  prepare  the  Model  Contract 
incorporating  the  numerous  negotiated  changes,  deliverables,  non  standard 
clauses  and  special  def inititions .  It  was  a  major  task  to  Insure  all  contrac- 
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tual  Language  resulting  from  over  two  months  of  Intensive  and  frequently 
concurrent  negotiations  was  properly  reflected  in  the  various  contracts.  The 
magnitude  of  the  effort  can  be  appreciated  when  one  considers  the  fact  that  each 
offeror's  Model  Contract  represented  a  stack  of  single  spaced  documents  over  2 
1/2  feet  high  and  each  page  had  to  be  reviewed  for  accuracy  by  various  groups. 

d.  Immediately  after  the  SAC  briefing  and  resultant  recommendation,  the 
Chairman,  Evaluation  Board  was  directed  to  personally  brief  the  key  DA  and  OSD 
decision  makers.  This  was  an  unusual  move  but  was  driven  by  time  constraints 
and  the  qualifications  of  the  Chairman.  These  numerous  briefings  were  presented 
on  a  highly  accelerated  schedule  with  virtually  no  time  for  graphics  preparation 
and  rehearsal.  In  fact  briefing  material  was  prepared  on  the  "fly",  a  high  risk 
endeavor  considering  the  level  of  briefing  (Secretary  of  Army,  Chief  of  Staff, 
Deputy  Secretary  of  Defense). 
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LESSONS  LEARNED 


1.  REQUIREMENT 

A.  In  order  Co  successfully  pursue  an  NDI  acquisition  strategy  a  non- 
system  specific  requirement  must  be  generated.  Most  requl resents  documents 
generated  today  are  too  system  specific  and  too  restrictive  in  system  perfor¬ 
mance  requirements,  thereby  eliminating  potential  bidders.  The  requirement  must 
be  stated  in  terms  of  vhat  the  systea  Is  to  do,  not  what  the  systea  Is.  The  MSE 
Operational  Capability  Oocuaent  (MSEOCD)  defined  five  functional  areas:  area 
coverage,  fixed  subscriber  access,  mobile  subscriber  access,  terminals  and 
system  control.  In  addition  to  these  five  functional  areas  there  were  a  number 
of  required  mandatory  priced  options.  All  bidders  were  required  to  bid  these 
priced  options.  The  last  section  of  the  MSEOCD  contained  numerous  desired  fea¬ 
tures.  Although  these  desired  features  were  not  an  absolute  requirement  the 
more  of  these  Incorporated  Into  the  offered  system  the  higher  the  evaluation 
rating.  The  MSEOCD  was  very  generic  In  that  it  defined  a  capability  but  left 
the  selection  of  the  equipment  to  provide  that  capability  to  the  offerors.  This 
document  became  the  cornerstone  of  the  MSE  acquisition  strategy.  The  Inability 
to  define  the  minimum  required  capability  can  be  a  "show  stopper". 


B.  Once  a  capability  has  been  defined  a  market  survey  must  be  conducted  to 


Insure  that  there  are  items  existing  that  can  satisfy  the  minimum  requirements 


systems  and  eliminate  til  of  them.  In  order  to  use  existing  systems,  and  avoid 
the  dollars  and  lead  time  associated  with  developing  a  new  system,  the  Army  must 


be  willing  to  compromise  on  non-essential  requirements.  The  Army  must  adjust 
doctrine,  if  necessary,  to  accommodate  the  existing  system  in  lieu  of  a  long 
development  program  aimed  at  acquiring  the  perfect  system.  The  user  must  be 
able  to  live  with  an  existing  system  if  that  is  the  only  system  bid.  In  the 
case  of  MSE  the  two  existing  systems  were  a  British  system  and  a  French  system. 


2.  SOLICITATION 

A.  The  request  for  proposal  must  be  generic  to  allow  maximum  competition. 
Although  this  may  seem  like  a  simple  concept  it  is  hard  to  Implement  within  the 
government  since  all  parties  Involved  are  used  to  preparing  standard  procurement 
data  packages  with  firm  MIL-STD  requirements.  In  order  to  Implement  an  MSE  NDI 
type  acquisition  the  request  for  proposals  must  be  tailored  to  allow  offerors  to 
describe  their  system  and  have  the  flexibility  to  propose  the  best  way  to  sup¬ 
port  and  use  the  system.  This  means  that  the  request  for  proposal  will  undergo 
several  revisions  to  insure  that  a  generic  package  Is  released  to  the  offerors. 
There  is  a  tendency  to  underestimate  the  time  required  for  this  effort.  Adequ¬ 
ate  time  should  be  allowed  to  prepare  the  request  for  proposal  and  senior  level 
involvement  is  a  must  to  settle  the  conflicts  that  will  inevitably  evolve. 


During  the  preparation  of  the  request  for  proposal  for  the  MSE  the  CG  of 
CECOM,  along  with  his  principal  directors,  personally  reviewed  the  draft  request 
for  proposal  to  insure  that  the  final  product  was  adequate  for  the  job  intended. 
Although  this  review  and  revision  adds  time  to  the  process  it  is  essential. 


The  panel  chiefs  and  the  principal  evaluators  oust  be  Involved  In  the 


preparation  of  the  request  for  proposal.  Although  this  step  requires  the 
Source  Selection  Evaluation  Board  to  renaln  In  session  for  an  extended  period  of 
time  It  Is  essential  that  these  individuals  understand  and  become  familiar  with 
the  entire  solicitation  package.  The  knowledge  gained  will  Insure  that  evalua- 
tors  are  used  in  an  effective  manner  and  the  panel  chief  will  be  in  a  position 
to  make  certain  that  the  proposals  are  fairly  and  completely  evaluated.  This 
up-front  effort  by  the  panel  chiefs  will  pay  high  dividends  at  a  later  date. 
Although  the  panel  chiefs  were  upgraded  prior  to  receipt  of  proposals  the  origi¬ 
nal  panel  chiefs  were  retained  as  deputies  to  the  new  panel  chiefs  and  this 
knowledge  was  preserved. 


C.  Common  ground  rules  must  be  established  prior  to  the  release  of  the 


solicltat Ion  In  such  areas  as  economic  price  adjustment,  currency  fluctuation, 
etc.  Establishing  these  ground  rules  prior  to  the  release  of  the  solicitation 
Is  essential  to  making  certain  that  all  offeror's  proposals  are  evaluated  fairly 
and  that  offerors  are  not  arbitrarily  eliminated.  Care  must  be  taken  in  estab¬ 
lishing  these  ground  rules  to  allow  offerors  to  make  business  decisions  that 
favor  his  bid.  Failure  to  establish  the  ground  rules  up-front  will  Increase  the 
complexity  of  the  evaluation  and  could  result  In  an  "apples  vs  oranges"  compari¬ 


son  especially  in  the  cost  arena. 


D.  Offerors  should  be  allowed  the  flexibility  to  adjust  the  quantities  of 


established  the  offerors  should  be  allowed  to  bid  the  types  and  quantities  to  be 
procured  In  a  given  year.  This  gives  the  offerors  the  flexibility  to  optimise 
their  production  rates  and  still  meet  the  needs  of  the  Army.  The  ability  to 
adjust  quantities  Is  also  Important  in  the  light  of  funding  requirements.  Small 
adjustments  in  procurement  quantities  may  be  necessary  to  meet  the  procurement 
appropriations  available. 


3.  SOURCE  SELECTION 


A.  The  need  for  administrative  support  cannot  be  overemphasized.  It  must 
be  remembered  that  a  Source  Selection  Evaluation  Board  Is  a  newly  created  separ¬ 
ate  organization  that  requires  the  same,  if  not  more,  administrative  support  as 
any  other  organization.  The  need  for  clerical,  audlo/vlsual,  reproduction, 
janitorial,  etc.,  support  can  be  critical.  The  MSE  evaluation  was  conducted  In 
inadequate  facilities  with  Inadequate  support  and  survived  only  through  the  ded¬ 
icated  effort  of  many  people.  It  is  unreasonable  to  expect  100+  people,  many  of 
them  on  temporary  duty,  to  work  7  days  a  week  in  marginal  facilities.  Adequate 
support  must  Include  word  processing  equipment,  graphics  support,  good  furni¬ 
ture,  experienced  secretaries,  reproduction  equipment  and  clean  facilities. 

B.  Receipt  of  proposals  is  the  start  point  for  negotiations.  This  goes 
against  the  traditional  way  of  doing  business  of  requiring  offerors  to  propose 
against  stringent  MIL-Standard  requirement  then  evaluating  the  proposals  against 
that  requirement.  Offerors  must  be  allowed  to  bid  what  they  feel  Is  their  best 
value  system  and  the  Army  must  evaluate  that  proposal  based  on  the  system's 
ability  to  meet  the  operational  requirement.  Deficiencies  noted  and  all  terms 
and  conditions  must  then  be  negotiated.  This  approach  allows  both  the  Army  and 


Che  offerors  Co  arrive  aC  an  agreed  upon  "best  value"  sysCea.  It  aust  always  be 


remembered  that  ic  is  Che  offeror's  proposal  and  he  has  Che  final  say  in  what  he 


is  bidding.  All  negotiations  aust  be  well  documented  and  there  aust  be  feedback 


between  panels.  The  minutes  of  all  negotiations  aust  be  published  quickly  as 


they  become  the  official  record  of  Che  negotiations  and  are  subject  to  audit  by 


both  sides.  One  of  the  dangers  in  having  concurrent  negotiations  is  that  one 


group  might  be  working  at  cross  purposes  to  another.  A  clear  negotiating  posi¬ 


tion  must  be  established  and  understood  by  all.  One  person,  in  the  case  of  HSE, 


the  contracting  officer,  aust  be  in  charge  of  all  negotiations.  All  panel  chiefs 


need  to  be  Instructed  in  how  to  negotiate  since  they  are  key  to  the  negotiation 


process  in  their  area  of  expertise.  It  should  be  understood  that  under  this 


acquisition  strategy  everything  is  open  to  negotiation,  to  Include  all  teras  and 


conditions  of  the  contract.  Open  dialogue  between  all  government  personnel  Is 


required  to  reach  closure  in  a  reasonable  tlae.  The  contracting  officer  must  be 


responsible  for  negotiating  package  deals  when  impasses  are  reached. 


C.  The  need  for  adequate  storage  facilities  cannot  be  overemphasized.  The 


solicitation  required  offerors  to  subait  all  existing  documentation,  such  as 


test  data,  in  support  of  their  proposals.  The  bids  arrived  in  tractor  trailers 


and  two  weeks  were  required  to  catalog  and  Inventory  the  proposals.  Soae  of  the 


data  was  valuable  but  auch  of  it  was  superfluous  such  as  copies  of  Aray  standard 


technical  manuals.  In  retrospect  it  would  have  been  wise  to  Instruct  the  offer¬ 


ors  to  list  the  supporting  docuaentatlon  available  and  request  only  that  data 


that  was  needed  by  the  evaluation  board. 
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D.  The  offerors  oust  be  allowed  to  bid  what  they  feel  is  their  best  value 
system.  There  Is  a  distinct  difference  between  the  best  operational  system  and 
the  best  value  system.  The  Government  should  strive  for  the  best  operational 
system  and  the  offerors  must  be  allowed  to  trade  off  operational  requirements 
and  cost.  The  best  and  final  offer  will  be  evaluated  on  how  well  it  meets  the 
operational  requirements  and  cost.  This  is  extremely  difficult  in  the  cost 
area,  and  was  not  put  to  the  test  during  the  NSE  evaluation  due  to  the  large 
cost  differences  between  the  two  offerors  proposed  price. 


E.  The  creation  of  model  contracts  to  reflect  what  was  negotiated  is  a 
major  task.  During  a  traditional  evaluation  the  request  for  proposal  contains 
all  the  requirements  and  is  the  base  line  for  all  negotiation  and  evaluation. 
During  the  evaluation  this  request  for  proposal  is  updated  to  reflect  all  agreed 
upon  changes  and  at  the  conclusion  of  negotiations  this  request  for  proposal 
becomes  the  contractual  document.  In  the  NSE  acquisition  a  model  contract  had 
to  be  built  from  scratch  for  each  offeror  since  the  equipment,  and  in  some  cases 
the  terras  and  conditions,  were  different.  This  became  a  herculean  task  since 
six  contracts  were  required,  three  for  each  offeror,  and  the  basic  contract 
alone  exceeded  five  thousand  pages.  The  administrative  burden  of  preparing 
these  model  contracts  can  easily  be  overlooked  during  this  evaluation  and  can 
become  a  delaying  factor  since  model  contracts  are  required  before  the  best  and 
final  offers  can  be  requested. 


The  number  of  contracts  should  be  kept  to  a  minimum  in  order  to  limit  the 


workload  required  while  still  maintaining  the  required  separation  between  con¬ 


tractual  Instruments.  The  MSE  program  required  three  contracts:  A.  basic 
contract  for  equipment  and  documentation,  a  requirements  contract  for  sustain¬ 
ment  efforts  and  replenishment  spares,  and  a  post  deployment  software  support 
contract . 


The  structure  of  all  contracts  should  be  carefully  examined  to  Insure  con¬ 
sistency  and  elimination  of  contradictory  clauses.  Avoid  price  ranges  since  they 
require  the  contractor  to  commit  to  an  Infinite  number  of  equipment  mixes  with 
an  Infinite  number  of  delivery  schedules.  The  MSE  contract  contains  a  delivery 
schedule  only  for  the  low  range  of  B. 


F.  Many  of  the  plans  and  other  data  required  update  as  a  result  of  negoti¬ 
ation  prior  to  Inclusion  In  the  model  contract.  Unfortunately  these  plans  were 
available  only  on  word  processing  equipment  at  each  of  the  offeror's  facilities. 
The  Government  should  direct  bidders  to  prepare  the  major  portion  of  their  pro 


osals  on  standard  word  processing  equipment  and  submit  the  source  media  with 


their  proposal.  This  would  eliminate  the  cut  and  pasting  associated  with  updat¬ 
ing  plans. 


G.  The  need  for  cost  data  depends  on  whether  adequate  competition  does  or 
does  not  exist.  During  the  MSE  evaluation  offerors  were  required  to  submit  cost 
proposals  along  with  their  original  proposals.  Because  of  the  limited  time 
allowed  to  submit  proposals  the  cost  proposals  were  budgetary  in  nature.  Hun¬ 
dreds  of  manhours  were  expended  in  evaluating  and  auditing  these  cost  proposals 
before  a  decision  was  made  that  adequate  competition  existed  and  cost  data  was 
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not  required.  This  issue  should  be  addressed  up  front  and  a  decision  made  prior 
to  the  first  submission  of  a  cost  proposal. 


H.  Bidders  must  be  allowed  to  modify  their  existing  equipment  to  meet  the 
requirement  to  prevent  arbitrary  elimination  of  candidate  systems.  Modification 
of  existing  equipment  should  be  minor  and/or  all  development  completed  prior  to 
contract  award  except  for  the  mandatory  priced  options.  All  modification  to  the 
baseline  system  must  be  evaluated  and  demonstrated  and  the  risk  assessed.  This 
is  one  of  the  most  difficult  tasks  required  during  the  evaluation  as  the  modifi¬ 
cations  cannot  be  judged  independently  of  each  other  but  must  be  related  to  the 
system  performance  as  a  whole. 


I.  Offerors  should  be  required  to  demonstrate  their  equipment  prior  to  the 
best  and  final  offer.  Three  overseas  demonstrations  were  required,  necessitating 
considerable  advanced  planning,  support  arrangements  and  in  some  cases  govern¬ 
ment  loaned  equipment.  The  purpose  of  the  demonstrations  was  to  provide 
confidence  that  the  offered  system  would  perform  as  outlined  in  the  proposal. 
Originally  the  RFP  specified  that  the  demonstrated  system  must  consist  of  only 
in  production  equipment  and  provide  for  a  certain  number  of  nodes,  static  sub¬ 
scriber  instruments  and  mobile  subscribers.  This  was  subsequently  modified  to 
allow  the  contractor  to  demonstrate  to  the  maximum  extent  possible  proposed  or 
existing  equipment. 


A  small  advance  party  consisting  of  legal,  contracting,  and  operational 
suitability  people  were  necessary  to  arrange  lodging,  transportation,  and  meals. 


It  was  decided  chat  executing  a  contract  to  cover  the  majority  of  the  basic 
costs  incurred  by  each  evaluator  while  TDY  was  much  easier  than  an  individual 


payment  method.  The  contracting  officer  must  be  prepared  to  execute  a  contract 
on  the  spot  with  civilian  hotels  and  transportation  firms. 

Planning  the  daily  demonstration  events  was  much  more  difficult  and  time 
consuming  than  originally  anticipated.  Because  of  the  large  number  of  evalua¬ 
tors,  limited  time,  and  the  necessity  to  evaluate  in  detail  specific  areas  of 
interest  a  complex,  time  phased  schedule  was  necessary.  Teams  were  established 
based  upon  area  of  evaluation  interest  with  OTEA  evaluators  integrated  Into 
these  teams.  Special  teams  were  constituted  consisting  of  executive  level  indi¬ 
viduals,  GAO  and  Army  Audit  Agency  representatives,  and  the  video  tape  crew.  A 
time  phased  daily  schedule  was  prepared  with  flexibility  provided  for  those 
unplanned  events  deemed  necessary  by  the  government.  In  sunnary,  the  EB's 
advance  party  essentially  designed  the  demonstration  and  in  one  Instance  dealt 
directly  with  allied  officers  (UK)  in  preparing  the  detailed  demonstration  sche¬ 
dule. 


The  GTE  demonstration  Incorporated  large  amounts  of  Government  owned 
equipment  that  had  to  be  shipped  from  many  locations,  modified  where  necessary 
by  GTE,  and  Incorporated  into  a  working  system.  Some  of  this  equipment  had  not 


been  fielded  and  extensive  arrangements  to  identify,  coordinate  a  release  from 
DA,  and  ship  the  equipment  in  a  compressed  time  schedule  was  necessary.  Over 
100  U.S.  soldiers  from  the  7th  Signal  Brigade,  Germany,  were  identified  to  set 
up  and  operate  the  U.S.  equipment.  Support  arrangements  had  to  be  coordinated 
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with  the  French  Army,  satellite  space  segment  obtained  and  USAREUR  HQ  fully 
briefed  on  the  movement  of  key  automatic  switches  from  Germany  to  France.  Reim¬ 
bursable  costs  for  the  U.S.  equipment  and  soldier  pay  had  to  be  negotiated  with 
GTE. 

Theater  clearance  and  travel  orders  became  a  major  issue  and  required  con¬ 
siderable  advance  planning  and  continual  emphasis  to  get  the  system  to  respond. 
In  order  to  save  a  few  dollars  per  individual,  the  EB  was  directed  to  use,  if 
available,  transporatlon  by  MAC  carrier.  MAC  space  was  booked  to  Frankfurt  for 
the  demonstration  in  France.  The  flight  was  cancelled  at  the  last  minute,  and 
it  was  impossible  to  book  commercial  transporatlon  for  the  large  contingent  as 
backup.  The  flight  went  as  scheduled  the  next  day  but  some  500  French  soldiers, 
100  GTE  representatives  and  100  U.S.  soldiers  were  delayed  a  day.  The  small 
cost  savings  is  not  a  substitute  for  close  adherence  to  a  tight  schedule. 

J.  Hand-in-hand  with  the  requirement  for  the  demonstration  is  the  fact 
that  operational  suitability  is  the  overriding  criteria.  It  must  be  made  per¬ 
fectly  clear  that  the  system  must  meet  the  operational  requirement  of  the  Army. 
The  operational  suitability  of  the  proposed  systems  should  be  evaluated  by  the 
actual  users.  The  MSE  operational  suitability  panel  chief  during  the  evaluation 
was  on  0-6  Signal  Corps  Brigade  Commander  supported  by  user  personnel  primarily 
from  the  U.S.  Army  Signal  Center,  Forces  Command,  and  various  TRADOC  posts.  It 
is  imperative  that  the  user,  represented  by  the  TRADOC  community,  be  the  final 
Judge  of  operational  suitability. 


K.  Interface  between  the  evaluation  panels  is  critical.  The  MSE  Evaluation 


Board  consisted  of  five  panels;  operational  suitability,  cost,  technical,  logis¬ 
tics,  and  management.  All  of  these  panels  make  decisions  which  are  critical  to 
the  other  panels,  especially  during  the  negotiation  phase.  Open  lines  of  commu¬ 
nications  were  established  and  maintained  by  daily  staff  meeting  chaired  by  the 
EB  Chairman  and  attended  by  the  panel  chiefs,  legal  advisor,  PM  advisor,  and 
contracting  officer.  One  of  the  factors  adversely  affecting  the  interface 
between  panels  was  the  physical  separation  of  the  panels  caused  by  the  fact  that 
the  Evaluation  Board  was  housed  in  four  separate  buildings.  The  entire  Board 
should  be  physically  located  in  one  building  when  possible.  Meetings  are  very 
important  but  must  be  structured  and  lengthy  dialog  between  small  groups  conduc¬ 
ted  off-line  to  avoid  wasting  time. 


L.  The  evaluation  is  subjective  since  the  criterium  is  the  system  that 
will  best  meet  the  Army's  requirement.  The  definition  of  "best"  is  hard  to 
define  in  quantifiable  terms.  All  offerors  must  understand  this  from  the 
outset.  This  was  communicated  to  all  potential  offerors  in  the  pre-bidder  con¬ 
ference  and  was  reiterated  to  the  offerors  throughout  the  negotiations. 

M.  Panel  chiefs  should  pre-brief  their  counterparts  on  the  SAC  prior  to 
each  SAC  meeting.  This  saves  time  during  the  actual  SAC  meeting  and  also  allows 
the  SAC  members  to  think  about  the  various  alternatives  and  decisions  to  be 
made.  The  drawback  is  that  each  panel  chief  must  travel  to  see  the  various  SAC 
members  prior  to  each  meeting.  A  realistic  schedule  must  include  time  for  these 
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4. 


GENERAL 


A.  The  MSE  NDI  acquisition  strategy  required  the  total  commitment  of  the 
Army  at  all  levels.  An  acquisition  of  this  scope  can  never  take  place  without 
the  total  commitment  from  all  levels.  There  are  many  examples  of  this  commit¬ 
ment:  The  Evaluation  Board  was  chaired  by  a  Major  General  assigned  to  TRADOC;  a 
Brigadier  General  was  assigned  as  the  Deputy  Chairman;  two  Brigade  Commanders 
were  assigned  as  the  Deputy  for  Operations  and  the  Operational  Suitability  Panel 
Chairman;  a  COL  was  assigned  as  the  Deputy  for  Systems  Engineering;  a  Senior 
Executive  Service  (SES)  grade  civilian  was  assigned  as  the  Chief  of  the  techni¬ 
cal  panel;  the  Senior  Advisory  Council  was  chaired  by  the  CG  of  the  Army 
Material  Command;  nine  General  Officers  and  five  senior  executive  service  mem¬ 
bers  served  on  the  Senior  Advisory  Council;  the  Under  Secretary  of  the  Army 
played  an  active  role  in  the  development  and  implementation  of  the  acquisition 
strategy;  the  Source  Selection  Authority  (SSA)  was  the  Secretary  of  the  Army. 


A  major  factor  was  the  composition  of  the  Senior  Advisory  Council.  The 
Senior  Advisory  Council  enjoyed  representation  from  all  Che  key  Army  elements 
involved  in  the  MSE  acquisition.  This  diverse  representation  allowed  the  Senior 
Advisory  Council  to  represent  the  total  Army  in  a  way  that  is  seldom  enjoyed  in 
the  acquisition  arena. 


B.  Although  it  is  easy  to  say  that  a  primary  ingredient  is  the  avoidance 


of  a  bureaucratic  mind  set,  it  is  hard  to  implement  in  a  bureaucratic  environ¬ 
ment.  We  are  all  captives  of  our  experience  and  most  of  the  players  in  the  MSE 
evaluation  have  spent  many  years  working  within  the  bureaucracy.  All  decisions 
should  be  subject  to  the  "common  sense"  test  as  opposed  to  that's  the  way  we 
have  always  done  it. 


C.  The  evaluation  should  include  all  key  agencies  and  organizations. 
Numerous  outside  agencies  were  allowed  to  monitor  the  evaluation  effort  (includ¬ 
ing  demonstrations)  creating  an  additional  burden  of  briefing  and  education. 
The  following  agencies  actively  participated  (part  time)  in  the  EB  in  primarily 
an  observer  status;  OTEA,  AMSAA,  GAO  and  the  AAA.  Although  these  personnel  re¬ 
quired  updates  and  education  their  participation  was  ultimately  of  significant 
value  in  "selling"  the  evaluation  through  DA  and  OSD.  Future  acquisitions 
should  consider  offering  similar  involvement  to  these  agencies. 


D.  An  Independent  Government  Cost  Estimate  (IGCE)  must  be  prepared  for 
each  offered  system  prior  to  best  and  final.  This  IGCE  is  used  to  establish 
price  reasonableness  and  is  essential  if  there  are  large  differences  in  the  cost 
of  the  various  systems.  These  IGCE's  were  essential  in  explaining  the  large 
cost  difference  between  the  two  offered  MSE  systems.  The  preparation  of  these 
IGCE's  is  a  time  consuming  task  and  must  be  completed  prior  to  receipt  of  best 


and  final  offers 


5.  ROLE  OF  THE  PROJECT  MANAGER 
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The  MSE  project  is  a  $4.3  billion  effort  to  provide  tactical  communication 
systems  at  Corps  and  Division  to  solve  command  and  control  communication  defi¬ 
ciencies  on  the  Air  Land  battlefield.  It  is  an  intensive  project  which  is 
expected  to  field  new  equipment  to  the  entire  tactical  force  structure  of  the 
Army,  both  active  and  reserve  components,  within  eight  years.  It  is  a  very 
ambitious  project  which  will  demand  intensive  project  management  of  the  highest 
order  if  the  schedule  is  to  be  maintained.  It  will  not  be  acceptable  to  solve 
problems  as  they  occur  because  that  will  be  too  late;  negative  trends  will  have 
to  be  perceived  as  they  start  and  corrective  measures  immediately  taken  to  stop 
the  problem  before  it  actually  occurs.  This  ability  to  recognize  incipient 
problems  must  exist  in  all  areas  pertinent  to  the  MSE  system  -  production,  test¬ 
ing,  training,  unit  reorganizations  within  the  Army,  MOS  realignments, 
facilities  preparation  for  resident  training  and  marshalling  and  fielding,  tran¬ 
sportation,  import  and  export  problems  and  a  myriad  of  like  items  involved  in 
any  and  all  worldwide  deployments.  It  is  important  to  note  that  several  of 
these  areas  of  interest  are  not  within  the  direct  purview  or  control  of  the  PM 
or  AMC.  The  MSE  system  is,  in  every  sense,  an  Army  project  that  must  be  addres¬ 
sed  and  pursued  in  a  timely  fashion  by  numerous  elements  in  the  Army  beyond  the 
development  community.  A  slip  in  any  of  these  elements  may  have  deleterious 
impacts  on  the  smooth  progression  of  MSE  to  a  successful  fielding  and  may  result 
in  Government  caused  delays  on  the  contract  with  resultant  cost  penalties. 


In  the  course  of  the  MSE  evaluation,  it  became  apparent  that  numerous  deci¬ 
sions  were  required  that  impacted  the  overall  project.  The  Government  had  asked 


industry  to  propose  a  system  on  a  turn-key  basis  which  included  training,  system 
support  to  include  support  structure  and  all  project  and  project  management 
aspects  of  producing,  testing,  delivering,  training,  fielding  and  supporting  the 
MSE  system  for  its  expected  lifetime  in  the  field.  Much  of  what  was  accomplish¬ 
ed  within  the  evaluation  will  have  a  lasting  impact  throughout  the  life  of  the 
MSE  system  in  the  Army.  The  evaluation,  with  its  extensive  negotiations,  was 
far  more  than  a  simple  evaluation  of  an  offer.  It  did  evaluate  the  offers,  but 
it  also  went  far,  through  negotiations,  in  defining  the  total  system  of  testing, 
acceptance,  support,  training,  business  terms  and  conditions,  and  project  man¬ 
agement  aspects  of  the  contractor  and,  hence,  the  Army  PM.  It  was,  therefore, 
much  more  than  a  pure  evaluation. 

It  is  essential  to  understand  that  the  evaluation  proceedings  of  a  proposal 
submitted  in  response  to  a  performance  specification  involving  more  than  the 
production  of  hardware,  must  address  a  number  of  sections  in  the  resultant  con¬ 
tract  or,  as  in  the  case  of  MSE,  actually  in  writing  almost  the  complete 
contract.  The  provisions  contained  in  the  contract  largely  define  what  is  to  be 
accomplished,  how  it  is  to  be  accomplished  and  the  checks  and  balances  to  be 
addressed  in  monitoring  and  supervising  the  progress  of  the  acquisition.  The 
contract  defines  the  program  to  a  large  degree  and  the  Project  Manager,  who  is 
responsible  for  the  accomplishment  of  the  program,  must  have  the  ability  to 
influence  the  writing  of  the  contract.  The  contract  must  be  executable  and  man¬ 
ageable.  While  this  is  obvious,  it  is  not  necessarily  of  great  concern  to  those 
personnel  who  are  tasked  to  evaluate  the  offer.  The  evaluators  task  is  to 
ascertain  strengths  and  weaknesses  and,  through  negotiation,  increase  the  former 
and  reduce  or  eliminate  the  latter.  The  actual  doing  of  what  is  cited  in  a  pro- 


posal  is  not  the  concern  of  the  evaluator;  that  belongs  to  the  Project  Manager. 
The  definition  of  the  offered  product  and  the  mechanisms  to  provide  that  product 
to  the  Government  are  evaluated  and,  as  appropriate,  negotiated  by  the  evalua¬ 
tor.  How  well  the  product  performs  and  how  well  the  mechanisms  of  production 
and  contractor  management  actually  work  is  the  responsibility  of  the  Project 
Manager. 

It  is  critical  that  the  Project  Manager  be  intimately  involved  in  any 
evaluation.  Where,  as  in  the  case  of  MSE  acquisition,  it  is  decided  to  raise 
the  source  selection  to  high  level/high  grade  participation  and  Involvement, 
care  must  be  exercised  to  assure  that  the  full  range  of  project /program  manage¬ 
ment  considerations  are  fully  provided  for  in  all  appointments.  Each 
appointment  should  be  carefully  weighed  against  the  task  to  be  accomplished,  the 
objective  sought  and  the  expertise  required  of  the  appointed  personnel.  The 
right  man  for  the  right  job  becomes  essential  since  numerous  decisions  are 
needed  to  achieve  the  desired  results.  If  such  decisions  were,  indeed,  not  re¬ 
quired,  then  the  need  to  raise  the  involvement  level  would  not  be  apparent 
either. 

An  evaluation  board  that  is  expected  to  conduct  broad  negotiations,  both 
technical  and  programmatic,  must  contain  a  prudent  mix  of  qualified  personnel  to 
address  each  area  under  consideration.  The  personnel  should  include  supervisory 
levels  from  the  Project  Manager's  office,  where  one  exists,  or  be  assigned  to 
the  Project  Manager's  office  where  one  is  to  be  established.  Such  assignments 
encourage  the  individuals  to  think  and  plan  beyond  the  evaluation  itself.  The 


Project  Manager  must  be  doing  his  planning  during  the  evaluation;  alternatives 


may  be  offered  during  the  course  of  an  evaluation  that  falls  clearly  within  the 
purview  of  the  Project  Manager  and  he,  or  his  close  subordinates,  should  have 
the  opportunity  to  influence  the  chosen  alternative  to  avoid  later  problems. 
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